Scipy -- Library of scientific algorithms for Python

Based on lecture at http://github.com/jrjohansson/scientific-python-lectures.

The SciPy framework builds on top of the low-level NumPy framework for multidimensional arrays, and provides a large number of higher-level scientific algorithms. Some of the topics that SciPy covers are:

• Special functions (scipy.special)
• Integration (scipy.integrate)
• Optimization (scipy.optimize)
• Interpolation (scipy.interpolate)
• Fourier Transforms (scipy.fftpack)
• Signal Processing (scipy.signal)
• Linear Algebra (scipy.linalg) [moved to numpy]
• Sparse Eigenvalue Problems (scipy.sparse)
• Statistics (scipy.stats)
• Multi-dimensional image processing (scipy.ndimage)
• File IO (scipy.io)

Each of these submodules provides a number of functions and classes that can be used to solve problems in their respective topics.

In this lecture we will look at how to use some of these subpackages.

To access the SciPy package in a Python program, we start by importing everything from the scipy module.

WARNING: In the new version of python many functionalities are now moved from scipy to numpy, but they are still available in scipy and a deprecated warning is displayed. The work-around is to first import functions from scipy and after that from numpy, to overwrite scipy functions with the same name.

from scipy import *
from numpy import *

If we only need to use part of the SciPy framework we can selectively include only those modules we are interested in. For example, to include the linear algebra package under the name la, we can do:

import scipy.linalg as la

Special functions

A large number of mathematical special functions are important for many computional physics problems. SciPy provides implementations of a very extensive set of special functions. For details, see the list of functions in the reference documention at http://docs.scipy.org/doc/scipy/reference/special.html#module-scipy.special.

To demonstrate the typical usage of special functions we will look in more detail at the Bessel functions:

from scipy import special
#help(special)

#
# The scipy.special module includes a large number of Bessel-functions
# Here we will use the functions jn and yn, which are the Bessel functions 
# of the first and second kind and real-valued order. We also include the 
# function jn_zeros and yn_zeros that gives the zeroes of the functions jn
# and yn.
#
from scipy.special import jn, yn, jn_zeros, yn_zeros

n = 0    # order
x = 0.0  # x-point

# Bessel function of first kind
print("J_%d(%f) = %f" % (n, x, jn(n, x)))

x = 1e-5
# Bessel function of second kind
print("Y_%d(%f) = %f" % (n, x, yn(n, x)))

J_0(0.000000) = 1.000000
Y_0(0.000010) = -7.403160

%matplotlib inline
import matplotlib.pyplot as plt
from IPython.display import Image

x = linspace(0, 10, 100)


for n in range(4):
    plt.plot(x, jn(n, x), label=('$J_%d(x)$' % n))
plt.legend(loc='best');

Second solution of the same Bessel differential equation: $$x^2 y_n''(x)+x y_n'(x)+(x^2-n^2) y_n(x)=0$$

Bessel functions of the first kind $J_n(x)$ are regular at the origin and $Y_n(x)$ are the solution which is irregular at the origin.

x = linspace(1, 10, 100)
for n in range(4):
    plt.plot(x, yn(n, x), label=('$J_%d(x)$' % n))
plt.legend(loc='best');

Integration

Numeric integration with general purpose quadrature

Numerical evaluation of a function of the type

$\displaystyle \int_a^b f(x) dx$

is called numerical quadrature, or simply quadature. SciPy provides a series of functions for different kind of quadrature, for example the quad, dblquad and tplquad for single, double and triple integrals, respectively.

from scipy.integrate import quad, dblquad, tplquad

The quad function takes a large number of optional arguments, which can be used to fine-tune the behaviour of the function (try help(quad) for details).

The basic usage is as follows:

x_lower, x_upper = 0.0, 1.0
val, abserr = quad( lambda x: x**2, x_lower, x_upper)
print('Value=', val, 'error=', abserr)

Value= 0.3333333333333333 error= 3.700743415417188e-15

# help(quad)

print( quad( lambda x: sin(x)/x, 0, 1000))

(1.5702669821983255, 0.24409510202674356)

/var/folders/j8/d9m3r0zx7j37l3ktfl_n1xw00000gn/T/ipykernel_94677/1866967918.py:1: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze 
  the integrand in order to determine the difficulties.  If the position of a 
  local difficulty can be determined (singularity, discontinuity) one will 
  probably gain from splitting up the interval and calling the integrator 
  on the subranges.  Perhaps a special-purpose integrator should be used.
  print( quad( lambda x: sin(x)/x, 0, 1000))

L=10*pi
x=linspace(1e-12,L,100)
plt.plot(x,sin(x)/x)
plt.grid()

If we need to pass extra arguments to integrand function we can use the args keyword argument. Let's say we want to evaluate

$f(x) = \displaystyle \int_0^x \frac{j_n(t)+j_m(t)}{t} dt$

# help(quad)

f2 = lambda t,n,m: (jn(n,t)+jn(m,t))/t

def f3(t,n,m):
    return (jn(n,t)+jn(m,t))/t

# First specific case for n=1, m=2, x=1
# integrating  (j_1(t)+j_2(t))/t
quad(f2, 0, 1, args=(1,2))

(0.5396292385998932, 5.991088054545437e-15)

Now simpler: $f_n(x) = n \displaystyle \int_0^x \frac{j_n(t)}{t} dt$

xs = linspace(1e-10,30,300) # mesh for x-variable

for ns in range(1,4): # n in 0,1,2,3
    fs=[ns * quad(lambda t,n: jn(n, t)/t, 0, x, args=(ns,))[0] for x in xs]
    plt.plot(xs,fs, label='n='+str(ns))
plt.legend(loc='best')

<matplotlib.legend.Legend at 0x165d58290>

# here we can get away withouth args since function is defined inline

for ns in range(1,4):
    fs=[ns * quad(lambda t: jn(ns, t)/t, 0, x)[0] for x in xs]
    plt.plot(xs,fs, label='n='+str(ns))
plt.legend(loc='best')

<matplotlib.legend.Legend at 0x165f2ded0>

Higher-dimensional integration works in the same way:

Note that we pass lambda functions for the limits for the y integration, since these in general can be functions of x.

dblquad: \begin{equation} \int_{x_a}^{x_b} dx \int_{y_a(x)}^{y_b(x)} dy f(x,y) \end{equation}

tplquad: \begin{equation} \int_{x_a}^{x_b} dx \int_{y_a(x)}^{y_b(x)} dy \int_{z_a(x,y)}^{z_b(x,y)} dz f(x,y,z) \end{equation}

def integrand(x, y, z):
    return exp(-x**2-y**2-z**2)

x_a,x_b = 0,10
y_a,y_b = 0,10
z_a,z_b = 0,10

val1, abserr = dblquad(integrand, x_a, x_b, lambda x:y_a, lambda x:y_b, args=(0,))

val2, abserr = tplquad(integrand, x_a, x_b, lambda x:y_a, lambda x:y_b, lambda x,y:z_a, lambda x,y:z_b)

print(val1, val2, abserr)

0.7853981633974476 0.6960409996039545 1.4675161390126584e-08

Ordinary differential equations (ODEs)

SciPy provides two different ways to solve ODEs: An API based on the function odeint, and object-oriented API based on the class ode. Usually odeint is easier to get started with, but the ode class offers some finer level of control.

Here we will use the odeint functions. For more information about the class ode, try help(ode). It does pretty much the same thing as odeint, but in an object-oriented fashion.

To use odeint, first import it from the scipy.integrate module

from scipy import *
from numpy import *
from scipy.integrate import odeint, ode

A system of ODEs should be formulated in standard form, which is:

$y' = f(y, t)$

where we are searching for function $y(t)$ and where

$y = [y_1(t), y_2(t), ..., y_n(t)]$

and $f$ is some function that gives the derivatives of the function $y_i(t)$. To solve an ODE we need to know the function $f$ and its initial condition, $y(0)$.

Note that higher-order ODEs can always be written in this form by introducing new variables for the intermediate derivatives.

For example, for second order differential equation $$y''(t)=f(y,y',t)$$ we can choose $$Y(t) = [ y(t), y'(t)]$$ and then $$\frac{dY(t)}{dt}=[y'(t),y''(t)]$$ which can be expressed as $$[\frac{dY_0(t)}{dt},\frac{dY_1(t)}{dt}]=[Y_1(t),f(Y_0(t),Y_1(t),t)].$$

Once we have defined the Python function f and array y_0 (that is $f$ and $y(0)$ in the mathematical formulation), we can use the odeint function as:

y_t = odeint(f, y_0, t)

where t is and array with time-coordinates for which to solve the ODE problem. y_t is an array with one row for each point in time in t, where each column corresponds to a solution y_i(t) at that point in time.

We will see how we can implement f and y_0 in Python code in the examples below.

Example: double pendulum

Let's consider a physical example: The double compound pendulum, described in some detail here: http://en.wikipedia.org/wiki/Double_pendulum

from IPython.display import Image
Image(url='http://upload.wikimedia.org/wikipedia/commons/c/c9/Double-compound-pendulum-dimensioned.svg')

The equations of motion of the pendulum are given on the wiki page:

${\dot \theta_1} = \frac{6}{m\ell^2} \frac{ 2 p_{\theta_1} - 3 \cos(\theta_1-\theta_2) p_{\theta_2}}{16 - 9 \cos^2(\theta_1-\theta_2)}$

${\dot \theta_2} = \frac{6}{m\ell^2} \frac{ 8 p_{\theta_2} - 3 \cos(\theta_1-\theta_2) p_{\theta_1}}{16 - 9 \cos^2(\theta_1-\theta_2)}.$

${\dot p_{\theta_1}} = -\frac{1}{2} m \ell^2 \left [ {\dot \theta_1} {\dot \theta_2} \sin (\theta_1-\theta_2) + 3 \frac{g}{\ell} \sin \theta_1 \right ]$

${\dot p_{\theta_2}} = -\frac{1}{2} m \ell^2 \left [ -{\dot \theta_1} {\dot \theta_2} \sin (\theta_1-\theta_2) + \frac{g}{\ell} \sin \theta_2 \right]$

To make the Python code simpler to follow, let's introduce new variable names and the vector notation: $x = [\theta_1, \theta_2, p_{\theta_1}, p_{\theta_2}]$

${\dot x_1} = \frac{6}{m\ell^2} \frac{ 2 x_3 - 3 \cos(x_1-x_2) x_4}{16 - 9 \cos^2(x_1-x_2)}$

${\dot x_2} = \frac{6}{m\ell^2} \frac{ 8 x_4 - 3 \cos(x_1-x_2) x_3}{16 - 9 \cos^2(x_1-x_2)}$

${\dot x_3} = -\frac{1}{2} m \ell^2 \left [ {\dot x_1} {\dot x_2} \sin (x_1-x_2) + 3 \frac{g}{\ell} \sin x_1 \right ]$

${\dot x_4} = -\frac{1}{2} m \ell^2 \left [ -{\dot x_1} {\dot x_2} \sin (x_1-x_2) + \frac{g}{\ell} \sin x_2 \right]$

# help(odeint)

def dx(x,t):
    """
    The right-hand side of the pendulum ODE
    x=[x1,x2,x3,x4]
    """
    g, L, m = 9.82, 1., 1.
    x1,x2,x3,x4 = x # x is array
    c1 = 1/(m*L**2)
    dx1 = 6.*c1 * (2*x3-3*cos(x1-x2)*x4)/(16.-9.*cos(x1-x2)**2)
    dx2 = 6.*c1 * (8*x4-3*cos(x1-x2)*x3)/(16.-9.*cos(x1-x2)**2)
    dx3 = -0.5/c1 * (dx1*dx2 * sin(x1-x2) + 3*g/L * sin(x1))
    dx4 = -0.5/c1 * (-dx1*dx2 * sin(x1-x2)+ g/L * sin(x2))
    return array([dx1,dx2,dx3,dx4])

# choose an initial state
x0 = [pi/4, pi/2, 0, 0]

# time coodinate to solve the ODE for: from 0 to 10 seconds
t = linspace(0, 10, 250)

# solve the ODE problem
x = odeint(dx, x0, t)

%matplotlib inline
import matplotlib.pyplot as plt


# plot the angles as a function of time
fig, axes = plt.subplots(1,2,figsize=(12,4))
axes[0].plot(t, x[:, 0], label="theta1")
axes[0].plot(t, x[:, 1], label="theta2")
axes[0].legend(loc='best')
axes[0].set_xlabel('time')

L = 0.5
x1 =  L * sin(x[:, 0])
y1 = -L * cos(x[:, 0])

x2 = x1 + L * sin(x[:, 1])
y2 = y1 - L * cos(x[:, 1])
    
axes[1].plot(x1, y1, label="pendulum1")
axes[1].plot(x2, y2, label="pendulum2")
axes[1].set_ylim([-1, 0])
axes[1].set_xlim([-1, 1])
axes[1].legend(loc='best');

examples for subplot creation: https://towardsdatascience.com/the-many-ways-to-call-axes-in-matplotlib-2667a7b06e06

matplotlib examples: https://matplotlib.org/3.1.1/gallery/index.html

See animation in pendulum.py.

(To get it work within jupyter seems a bit challenging at the moment.)

Example: Damped harmonic oscillator

ODE problems are important in computational physics, so we will look at one more example: the damped harmonic oscillation. This problem is well described on the wiki page: http://en.wikipedia.org/wiki/Damping

The equation of motion for the damped oscillator is:

$\displaystyle \frac{\mathrm{d}^2x}{\mathrm{d}t^2} + 2\zeta\omega_0\frac{\mathrm{d}x}{\mathrm{d}t} + \omega^2_0 x = 0$

where $x$ is the position of the oscillator, $\omega_0$ is the frequency, and $\zeta$ is the damping ratio. To write this second-order ODE on standard form we introduce $p = \frac{\mathrm{d}x}{\mathrm{d}t}$:

$\displaystyle \frac{\mathrm{d}p}{\mathrm{d}t} = - 2\zeta\omega_0 p - \omega^2_0 x$

$\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t} = p$

In the implementation of this example we will add extra arguments to the RHS function for the ODE, rather than using global variables as we did in the previous example. As a consequence of the extra arguments to the RHS, we need to pass an keyword argument args to the odeint function:

def dy(y, t, zeta, w0):
    """
    The right-hand side of the damped oscillator ODE
    """
    x, p = y #y[0], y[1]
    
    dx = p
    dp = -2 * zeta * w0 * p - w0**2 * x

    return array([dx, dp])

from numpy import *

# initial state: 
y0 = [1.0, 0.0]

# time coodinate to solve the ODE for
t = linspace(0, 10, 1000)
w0 = 2*pi*1.0

from scipy.integrate import odeint

# solve the ODE problem for three different values of the damping ratio

y1 = odeint(dy, y0, t, args=(0.0, w0)) # undamped
y2 = odeint(dy, y0, t, args=(0.2, w0)) # under damped
y3 = odeint(dy, y0, t, args=(1.0, w0)) # critial damping
y4 = odeint(dy, y0, t, args=(5.0, w0)) # over damped

import pylab as plt

plt.plot(t, y1[:,0], label="undamped", linewidth=0.5)
plt.plot(t, y2[:,0], label="under damped")
plt.plot(t, y3[:,0], label=r"critical damping")
plt.plot(t, y4[:,0], label="over damped")
plt.legend();

Fourier transform

Fourier transforms are one of the universal tools in computational physics, which appear over and over again in different contexts: solving partial DE, quantum mechanics, etc.

SciPy provides functions for accessing the classic FFTPACK library from NetLib, which is an efficient and well tested FFT library written in FORTRAN. The SciPy API has a few additional convenience functions, but overall the API is closely related to the original FORTRAN library.

Mathematically Fourier transfrom is an operation that produces frequency distribution of a function: \begin{eqnarray} F(\omega) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{i\omega t} f(t) dt \end{eqnarray}

One way to picture this operation is to think of a sound wave: The Fourier transform is analogous to decomposing the sound of a musical chord into the intensities of its constituent pitches.

The important property is that its inverse exists and gives back the original function:

\begin{eqnarray} f(t) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-i\omega t} F(\omega) d\omega \end{eqnarray}

The proof is relatively straighforward. We insert expression for $F(\omega)$ into the second equation, and get \begin{eqnarray} f(t) = \frac{1}{2\pi} \int_{-\infty}^{\infty}d\omega e^{-i\omega t} \int_{-\infty}^{\infty} dt' e^{i\omega t'} f(t')=\frac{1}{2\pi} \int_{-\infty}^{\infty}dt' f(t')\int_{-\infty}^{\infty}d\omega e^{i\omega (t'-t)} \end{eqnarray}

Next we will prove that \begin{eqnarray} \frac{1}{2\pi}\int_{-\infty}^{\infty}d\omega e^{i\omega (t'-t)} =\delta(t-t') \end{eqnarray} which is the Kronecker $\delta$ function, and has property that \begin{equation} \int_{-\infty}^{\infty}dt' f(t')\delta(t-t')=f(t) \end{equation}

The properies of the Kronecker $\delta(\tau)$ functions are that for a small $\varepsilon$: \begin{eqnarray} && \delta(\tau)=0 ; |\tau|>\varepsilon \\ && \int_{-\varepsilon}^{\varepsilon}\delta(\tau) = 1 \end{eqnarray}

We need to prove that \begin{eqnarray} \frac{1}{2\pi}\int_{-\infty}^{\infty}d\omega e^{i\omega \tau} =\delta(\tau) \end{eqnarray}

If $\tau$ is nonzero, the value is vanishig as long as the limits in $\infty$ are properly arranged. The second property of $\delta$ function still needs to be proven: \begin{eqnarray} &&\frac{1}{2\pi}\int_{-\varepsilon}^{\varepsilon} d\tau \int_{-\infty}^{\infty}d\omega e^{i\omega \tau} =^? 1\\ &&\frac{1}{2\pi} \int_{-\infty}^{\infty}d\omega \int_{-\varepsilon}^{\varepsilon} d\tau e^{i\omega \tau}= \frac{1}{2\pi} \int_{-\infty}^{\infty}d\omega \frac{e^{i\omega\epsilon}-e^{-i\omega\epsilon}}{i\omega}= \frac{1}{2\pi} \int_{-\infty}^{\infty}d\omega \frac{2 i \sin(\omega\epsilon)}{i\omega}= \frac{1}{\pi} \int_{-\infty}^{\infty}d x \frac{\sin(x)}{x}=1 \end{eqnarray}

from math import *
from scipy import integrate

r=integrate.quad(lambda x: sin(x)/x, 1e-6,200.)
r[0]/pi

0.4992312856179283

from scipy import special
# help(special.sici)

from scipy import special
special.sici(100000.)[0]/pi

0.5000031810631101

Note that Fourier transform of a real function is in general a complex function because

\begin{eqnarray} \textrm{Im}(F(\omega)) = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty} \sin(\omega t) f(t)dt \end{eqnarray}

Only if $f(t)$ is even function of $t$ the imaginary part of $F$ vanishes, and $F(\omega)$ is real.

If $f(t)$ is a real function, than $F(\omega)$ has to satisfy the constrain:

\begin{eqnarray} F^*(\omega) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-i\omega t} f(t) dt = F(-\omega) \end{eqnarray}

which means that the real part $Re F(\omega)= Re F(-\omega)$ is even, and $Im F(\omega)=-Im F(-\omega)$ is odd.

Periodic functions with period T

Note also that periodic functions have descrete frequency only, i.e., $F(\omega)$ is nonzero only for discrete set of numbers (discrete function).

The best way to see this is to request that function $f(t)$ is perodic and hence $f(t+T)=f(t)$ where $T$ is a period. Than we know that \begin{eqnarray} f(t) = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^\infty d\omega F(\omega) e^{-i\omega t}=f(t+T)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^\infty d\omega F(\omega) e^{-i\omega t-i\omega T} \end{eqnarray}

and for this to be true, we must have $\omega T = 2\pi n$, or, $\omega_n = 2\pi n/T$. This means that frequency is descrete and only values $\omega_n=2\pi n/T$ are allowed, where $n$ is integer. If $T$ is large, there are still a lot of freuquencies in every small interval allowed.

For discrete case, we usually choose different normalization of functions. Instead of $F(\omega_n)$ we can use more appropriate normalization $\widetilde{F}(\omega_n)$ such that

\begin{eqnarray} f(t) = \frac{1}{\sqrt{T}}\sum_n \widetilde{F}(\omega_n)e^{-i\omega_n t}\\ \widetilde{F}(\omega_n) = \frac{1}{\sqrt{T}}\int_{t_0}^{t_0+T} f(t) e^{i\omega_n t} dt \end{eqnarray}

We can prove that this is correct normalization, because $$\frac{1}{T}\int_{t_0}^{t_0+T}dt e^{i 2\pi(n-n')\frac{t}{T}} = \delta_{n-n'}$$

Discretized functions for practical FFT

Finally, in practice we discretize the funtion \begin{eqnarray} f(t) = \frac{1}{N}\sum_{j=0}^{N-1} \delta(t-t_j) f(t_j) \end{eqnarray}

If we rename $f(t_j)\equiv f_j$ and $F(\omega_n)\equiv F_n$ we arrive at the discrete Fourier transform.

For the variable $\frac{t}{T}$, we choosen to be disributed between $[0,1]$ ($t_0$ is set to 0). For a number distributed between $[0,1]$ we can choose enumeration $j/N$, where $j\in [0,N-1]$, i.e, $t = T j/N$ with $N$ being a large number. Then $\omega_n t_j= 2\pi n\; t_j/T = 2\pi n j/N$. The discrete Fourier transform thus takes the form \begin{eqnarray} && F_n = \frac{1}{\sqrt{N}}\sum_{j=0}^{N-1} f_j e^{i 2\pi n j/N}\\ && f_j = \frac{1}{\sqrt{N}} \sum_{n=0}^{N-1} F_n e^{-i 2\pi n j/N} \end{eqnarray}

The unusal feature of FFT algorithm is that it can be done in $O(N \log(N))$ time, rather than $O(N^2)$ time, as naively estimated from the above equation.

In QM position ($x$) and momentum($p$) are connected by Fourier transform. Similarly time $t$ and energy $E=\omega/\hbar$ are connected by Fourier transform.

Fourier transform is also used in solving partial differential equations.

Some examples displayed below:

To use the fftpack module in a python program, include it using:

from scipy.fftpack import *
from numpy.fft import fftfreq

To demonstrate how to do a fast Fourier transform with SciPy, let's look at the FFT of the solution to the damped oscillator from the previous section:

# calculate the fast fourier transform
# y2 is the solution to the under-damped oscillator from the previous section
F = fft(y2[:,0]) 

# calculate the frequencies for the components in F
w = fftfreq(len(t), t[1]-t[0])

T = len(t)*(t[1]-t[0])
print('Period T={:f}  distance domega= 1/T={:f} last point=(N/2-1)/T={:f} first point=-N/(2T)={:f}'.format(
    T, 1/T, (len(t)/2-1)*1/T, -len(t)/(2*T) ))

Period T=10.010010  distance domega= 1/T=0.099900 last point=(N/2-1)/T=49.850100 first point=-N/(2T)=-49.950000

# w_n=n/(len(t)*(t[1]-t[0])) == n/T
# note that analytically we expect w_n = 2*pi * n/T

print(range(10)/(len(t)*(t[1]-t[0])))
w

[0.     0.0999 0.1998 0.2997 0.3996 0.4995 0.5994 0.6993 0.7992 0.8991]

array([  0.    ,   0.0999,   0.1998,   0.2997,   0.3996,   0.4995,
         0.5994,   0.6993,   0.7992,   0.8991,   0.999 ,   1.0989,
         1.1988,   1.2987,   1.3986,   1.4985,   1.5984,   1.6983,
         1.7982,   1.8981,   1.998 ,   2.0979,   2.1978,   2.2977,
         2.3976,   2.4975,   2.5974,   2.6973,   2.7972,   2.8971,
         2.997 ,   3.0969,   3.1968,   3.2967,   3.3966,   3.4965,
         3.5964,   3.6963,   3.7962,   3.8961,   3.996 ,   4.0959,
         4.1958,   4.2957,   4.3956,   4.4955,   4.5954,   4.6953,
         4.7952,   4.8951,   4.995 ,   5.0949,   5.1948,   5.2947,
         5.3946,   5.4945,   5.5944,   5.6943,   5.7942,   5.8941,
         5.994 ,   6.0939,   6.1938,   6.2937,   6.3936,   6.4935,
         6.5934,   6.6933,   6.7932,   6.8931,   6.993 ,   7.0929,
         7.1928,   7.2927,   7.3926,   7.4925,   7.5924,   7.6923,
         7.7922,   7.8921,   7.992 ,   8.0919,   8.1918,   8.2917,
         8.3916,   8.4915,   8.5914,   8.6913,   8.7912,   8.8911,
         8.991 ,   9.0909,   9.1908,   9.2907,   9.3906,   9.4905,
         9.5904,   9.6903,   9.7902,   9.8901,   9.99  ,  10.0899,
        10.1898,  10.2897,  10.3896,  10.4895,  10.5894,  10.6893,
        10.7892,  10.8891,  10.989 ,  11.0889,  11.1888,  11.2887,
        11.3886,  11.4885,  11.5884,  11.6883,  11.7882,  11.8881,
        11.988 ,  12.0879,  12.1878,  12.2877,  12.3876,  12.4875,
        12.5874,  12.6873,  12.7872,  12.8871,  12.987 ,  13.0869,
        13.1868,  13.2867,  13.3866,  13.4865,  13.5864,  13.6863,
        13.7862,  13.8861,  13.986 ,  14.0859,  14.1858,  14.2857,
        14.3856,  14.4855,  14.5854,  14.6853,  14.7852,  14.8851,
        14.985 ,  15.0849,  15.1848,  15.2847,  15.3846,  15.4845,
        15.5844,  15.6843,  15.7842,  15.8841,  15.984 ,  16.0839,
        16.1838,  16.2837,  16.3836,  16.4835,  16.5834,  16.6833,
        16.7832,  16.8831,  16.983 ,  17.0829,  17.1828,  17.2827,
        17.3826,  17.4825,  17.5824,  17.6823,  17.7822,  17.8821,
        17.982 ,  18.0819,  18.1818,  18.2817,  18.3816,  18.4815,
        18.5814,  18.6813,  18.7812,  18.8811,  18.981 ,  19.0809,
        19.1808,  19.2807,  19.3806,  19.4805,  19.5804,  19.6803,
        19.7802,  19.8801,  19.98  ,  20.0799,  20.1798,  20.2797,
        20.3796,  20.4795,  20.5794,  20.6793,  20.7792,  20.8791,
        20.979 ,  21.0789,  21.1788,  21.2787,  21.3786,  21.4785,
        21.5784,  21.6783,  21.7782,  21.8781,  21.978 ,  22.0779,
        22.1778,  22.2777,  22.3776,  22.4775,  22.5774,  22.6773,
        22.7772,  22.8771,  22.977 ,  23.0769,  23.1768,  23.2767,
        23.3766,  23.4765,  23.5764,  23.6763,  23.7762,  23.8761,
        23.976 ,  24.0759,  24.1758,  24.2757,  24.3756,  24.4755,
        24.5754,  24.6753,  24.7752,  24.8751,  24.975 ,  25.0749,
        25.1748,  25.2747,  25.3746,  25.4745,  25.5744,  25.6743,
        25.7742,  25.8741,  25.974 ,  26.0739,  26.1738,  26.2737,
        26.3736,  26.4735,  26.5734,  26.6733,  26.7732,  26.8731,
        26.973 ,  27.0729,  27.1728,  27.2727,  27.3726,  27.4725,
        27.5724,  27.6723,  27.7722,  27.8721,  27.972 ,  28.0719,
        28.1718,  28.2717,  28.3716,  28.4715,  28.5714,  28.6713,
        28.7712,  28.8711,  28.971 ,  29.0709,  29.1708,  29.2707,
        29.3706,  29.4705,  29.5704,  29.6703,  29.7702,  29.8701,
        29.97  ,  30.0699,  30.1698,  30.2697,  30.3696,  30.4695,
        30.5694,  30.6693,  30.7692,  30.8691,  30.969 ,  31.0689,
        31.1688,  31.2687,  31.3686,  31.4685,  31.5684,  31.6683,
        31.7682,  31.8681,  31.968 ,  32.0679,  32.1678,  32.2677,
        32.3676,  32.4675,  32.5674,  32.6673,  32.7672,  32.8671,
        32.967 ,  33.0669,  33.1668,  33.2667,  33.3666,  33.4665,
        33.5664,  33.6663,  33.7662,  33.8661,  33.966 ,  34.0659,
        34.1658,  34.2657,  34.3656,  34.4655,  34.5654,  34.6653,
        34.7652,  34.8651,  34.965 ,  35.0649,  35.1648,  35.2647,
        35.3646,  35.4645,  35.5644,  35.6643,  35.7642,  35.8641,
        35.964 ,  36.0639,  36.1638,  36.2637,  36.3636,  36.4635,
        36.5634,  36.6633,  36.7632,  36.8631,  36.963 ,  37.0629,
        37.1628,  37.2627,  37.3626,  37.4625,  37.5624,  37.6623,
        37.7622,  37.8621,  37.962 ,  38.0619,  38.1618,  38.2617,
        38.3616,  38.4615,  38.5614,  38.6613,  38.7612,  38.8611,
        38.961 ,  39.0609,  39.1608,  39.2607,  39.3606,  39.4605,
        39.5604,  39.6603,  39.7602,  39.8601,  39.96  ,  40.0599,
        40.1598,  40.2597,  40.3596,  40.4595,  40.5594,  40.6593,
        40.7592,  40.8591,  40.959 ,  41.0589,  41.1588,  41.2587,
        41.3586,  41.4585,  41.5584,  41.6583,  41.7582,  41.8581,
        41.958 ,  42.0579,  42.1578,  42.2577,  42.3576,  42.4575,
        42.5574,  42.6573,  42.7572,  42.8571,  42.957 ,  43.0569,
        43.1568,  43.2567,  43.3566,  43.4565,  43.5564,  43.6563,
        43.7562,  43.8561,  43.956 ,  44.0559,  44.1558,  44.2557,
        44.3556,  44.4555,  44.5554,  44.6553,  44.7552,  44.8551,
        44.955 ,  45.0549,  45.1548,  45.2547,  45.3546,  45.4545,
        45.5544,  45.6543,  45.7542,  45.8541,  45.954 ,  46.0539,
        46.1538,  46.2537,  46.3536,  46.4535,  46.5534,  46.6533,
        46.7532,  46.8531,  46.953 ,  47.0529,  47.1528,  47.2527,
        47.3526,  47.4525,  47.5524,  47.6523,  47.7522,  47.8521,
        47.952 ,  48.0519,  48.1518,  48.2517,  48.3516,  48.4515,
        48.5514,  48.6513,  48.7512,  48.8511,  48.951 ,  49.0509,
        49.1508,  49.2507,  49.3506,  49.4505,  49.5504,  49.6503,
        49.7502,  49.8501, -49.95  , -49.8501, -49.7502, -49.6503,
       -49.5504, -49.4505, -49.3506, -49.2507, -49.1508, -49.0509,
       -48.951 , -48.8511, -48.7512, -48.6513, -48.5514, -48.4515,
       -48.3516, -48.2517, -48.1518, -48.0519, -47.952 , -47.8521,
       -47.7522, -47.6523, -47.5524, -47.4525, -47.3526, -47.2527,
       -47.1528, -47.0529, -46.953 , -46.8531, -46.7532, -46.6533,
       -46.5534, -46.4535, -46.3536, -46.2537, -46.1538, -46.0539,
       -45.954 , -45.8541, -45.7542, -45.6543, -45.5544, -45.4545,
       -45.3546, -45.2547, -45.1548, -45.0549, -44.955 , -44.8551,
       -44.7552, -44.6553, -44.5554, -44.4555, -44.3556, -44.2557,
       -44.1558, -44.0559, -43.956 , -43.8561, -43.7562, -43.6563,
       -43.5564, -43.4565, -43.3566, -43.2567, -43.1568, -43.0569,
       -42.957 , -42.8571, -42.7572, -42.6573, -42.5574, -42.4575,
       -42.3576, -42.2577, -42.1578, -42.0579, -41.958 , -41.8581,
       -41.7582, -41.6583, -41.5584, -41.4585, -41.3586, -41.2587,
       -41.1588, -41.0589, -40.959 , -40.8591, -40.7592, -40.6593,
       -40.5594, -40.4595, -40.3596, -40.2597, -40.1598, -40.0599,
       -39.96  , -39.8601, -39.7602, -39.6603, -39.5604, -39.4605,
       -39.3606, -39.2607, -39.1608, -39.0609, -38.961 , -38.8611,
       -38.7612, -38.6613, -38.5614, -38.4615, -38.3616, -38.2617,
       -38.1618, -38.0619, -37.962 , -37.8621, -37.7622, -37.6623,
       -37.5624, -37.4625, -37.3626, -37.2627, -37.1628, -37.0629,
       -36.963 , -36.8631, -36.7632, -36.6633, -36.5634, -36.4635,
       -36.3636, -36.2637, -36.1638, -36.0639, -35.964 , -35.8641,
       -35.7642, -35.6643, -35.5644, -35.4645, -35.3646, -35.2647,
       -35.1648, -35.0649, -34.965 , -34.8651, -34.7652, -34.6653,
       -34.5654, -34.4655, -34.3656, -34.2657, -34.1658, -34.0659,
       -33.966 , -33.8661, -33.7662, -33.6663, -33.5664, -33.4665,
       -33.3666, -33.2667, -33.1668, -33.0669, -32.967 , -32.8671,
       -32.7672, -32.6673, -32.5674, -32.4675, -32.3676, -32.2677,
       -32.1678, -32.0679, -31.968 , -31.8681, -31.7682, -31.6683,
       -31.5684, -31.4685, -31.3686, -31.2687, -31.1688, -31.0689,
       -30.969 , -30.8691, -30.7692, -30.6693, -30.5694, -30.4695,
       -30.3696, -30.2697, -30.1698, -30.0699, -29.97  , -29.8701,
       -29.7702, -29.6703, -29.5704, -29.4705, -29.3706, -29.2707,
       -29.1708, -29.0709, -28.971 , -28.8711, -28.7712, -28.6713,
       -28.5714, -28.4715, -28.3716, -28.2717, -28.1718, -28.0719,
       -27.972 , -27.8721, -27.7722, -27.6723, -27.5724, -27.4725,
       -27.3726, -27.2727, -27.1728, -27.0729, -26.973 , -26.8731,
       -26.7732, -26.6733, -26.5734, -26.4735, -26.3736, -26.2737,
       -26.1738, -26.0739, -25.974 , -25.8741, -25.7742, -25.6743,
       -25.5744, -25.4745, -25.3746, -25.2747, -25.1748, -25.0749,
       -24.975 , -24.8751, -24.7752, -24.6753, -24.5754, -24.4755,
       -24.3756, -24.2757, -24.1758, -24.0759, -23.976 , -23.8761,
       -23.7762, -23.6763, -23.5764, -23.4765, -23.3766, -23.2767,
       -23.1768, -23.0769, -22.977 , -22.8771, -22.7772, -22.6773,
       -22.5774, -22.4775, -22.3776, -22.2777, -22.1778, -22.0779,
       -21.978 , -21.8781, -21.7782, -21.6783, -21.5784, -21.4785,
       -21.3786, -21.2787, -21.1788, -21.0789, -20.979 , -20.8791,
       -20.7792, -20.6793, -20.5794, -20.4795, -20.3796, -20.2797,
       -20.1798, -20.0799, -19.98  , -19.8801, -19.7802, -19.6803,
       -19.5804, -19.4805, -19.3806, -19.2807, -19.1808, -19.0809,
       -18.981 , -18.8811, -18.7812, -18.6813, -18.5814, -18.4815,
       -18.3816, -18.2817, -18.1818, -18.0819, -17.982 , -17.8821,
       -17.7822, -17.6823, -17.5824, -17.4825, -17.3826, -17.2827,
       -17.1828, -17.0829, -16.983 , -16.8831, -16.7832, -16.6833,
       -16.5834, -16.4835, -16.3836, -16.2837, -16.1838, -16.0839,
       -15.984 , -15.8841, -15.7842, -15.6843, -15.5844, -15.4845,
       -15.3846, -15.2847, -15.1848, -15.0849, -14.985 , -14.8851,
       -14.7852, -14.6853, -14.5854, -14.4855, -14.3856, -14.2857,
       -14.1858, -14.0859, -13.986 , -13.8861, -13.7862, -13.6863,
       -13.5864, -13.4865, -13.3866, -13.2867, -13.1868, -13.0869,
       -12.987 , -12.8871, -12.7872, -12.6873, -12.5874, -12.4875,
       -12.3876, -12.2877, -12.1878, -12.0879, -11.988 , -11.8881,
       -11.7882, -11.6883, -11.5884, -11.4885, -11.3886, -11.2887,
       -11.1888, -11.0889, -10.989 , -10.8891, -10.7892, -10.6893,
       -10.5894, -10.4895, -10.3896, -10.2897, -10.1898, -10.0899,
        -9.99  ,  -9.8901,  -9.7902,  -9.6903,  -9.5904,  -9.4905,
        -9.3906,  -9.2907,  -9.1908,  -9.0909,  -8.991 ,  -8.8911,
        -8.7912,  -8.6913,  -8.5914,  -8.4915,  -8.3916,  -8.2917,
        -8.1918,  -8.0919,  -7.992 ,  -7.8921,  -7.7922,  -7.6923,
        -7.5924,  -7.4925,  -7.3926,  -7.2927,  -7.1928,  -7.0929,
        -6.993 ,  -6.8931,  -6.7932,  -6.6933,  -6.5934,  -6.4935,
        -6.3936,  -6.2937,  -6.1938,  -6.0939,  -5.994 ,  -5.8941,
        -5.7942,  -5.6943,  -5.5944,  -5.4945,  -5.3946,  -5.2947,
        -5.1948,  -5.0949,  -4.995 ,  -4.8951,  -4.7952,  -4.6953,
        -4.5954,  -4.4955,  -4.3956,  -4.2957,  -4.1958,  -4.0959,
        -3.996 ,  -3.8961,  -3.7962,  -3.6963,  -3.5964,  -3.4965,
        -3.3966,  -3.2967,  -3.1968,  -3.0969,  -2.997 ,  -2.8971,
        -2.7972,  -2.6973,  -2.5974,  -2.4975,  -2.3976,  -2.2977,
        -2.1978,  -2.0979,  -1.998 ,  -1.8981,  -1.7982,  -1.6983,
        -1.5984,  -1.4985,  -1.3986,  -1.2987,  -1.1988,  -1.0989,
        -0.999 ,  -0.8991,  -0.7992,  -0.6993,  -0.5994,  -0.4995,
        -0.3996,  -0.2997,  -0.1998,  -0.0999])

plt.plot(w, abs(F));
plt.xlim([-5,5])

(-5.0, 5.0)

Frequencies are sorted in inconvenient way. First is $\omega_n=0$, followed by all positive frequencies, and than followed by all negative frequencies. The frequnecy thus jumps from $\frac{(N-2)}{2 T}$ to $-\frac{N}{2 T}$. The reason for that is efficiency of FFT routine.

To make sense of this output, we sort the frequencies in ascending order:

indx=sorted(range(len(w)),key=lambda i: w[i])

The frequencies are also missing $2\pi$ from our definitions. The discrete frequencies should be $2\pi n/T$, but the program gives us $n/T$, with $n$ between $-N/2$ to $N/2-1$ (for even $N$).

ws = 2*pi*w[indx]
Fs = F[indx]

plt.plot(ws, abs(Fs));
plt.plot(ws, Fs.real);
plt.plot(ws, Fs.imag);
plt.xlim([-30,30])

(-30.0, 30.0)

Properties of Fourier transform of a real signal: \begin{eqnarray} && F(\omega) = \int e^{i\omega t} x(t) dt\\ && F^*(\omega) = F(-\omega)\\ && Re(F(\omega)) = Re(F(-\omega))\\ && Im(F(\omega)) = -Im(F(-\omega)) \end{eqnarray}

The spectrum is peaked at $2\pi$, because our frequency $\omega_0=2\pi$. The damping adds a lot of additional frequencies to the spectrum. The spectrum has correct properties.

Can we use the inverse Fourier transform to get the original spectra back?

Inverse Fourier is obtained by ifft.

ft = ifft(Fs)

ts = fftfreq(len(ws), ws[1]-ws[0])*2*pi

plt.plot(ts,ft.real)
plt.xlim([-5,5])
plt.plot(t,y2[:,0]);

It seems similar, but something seems wrong.... Every second point is correct, but every second point has exactly minus sign as compared to expectations. Why?

plt.plot(ts,ft.real)
plt.xlim([0,0.2])
plt.plot(t,y2[:,0])

[<matplotlib.lines.Line2D at 0x16623e3d0>]

The issue is that if variable $t$ starts with 0 and extends to $T$, than frequency should also start with $\omega=0$ and end with $\omega_{max}$. Alternatively, we could shift all functions to start with $t\in[-T/2,T/2]$ and $\omega=[-\omega_{max}/2,\omega_{max}/2]$.

What we expect from FFT is \begin{eqnarray} f(t_i) = \frac{1}{\sqrt{N}} \sum_{n=-N/2}^{n=N/2-1} e^{-i\omega_n t_i} F_n \end{eqnarray} where $\omega_n=2\pi n/T$. But what FFT does is \begin{eqnarray} f(t_i)=\frac{1}{\sqrt{N}} \sum_{n=0}^{N-1} e^{-i\omega_n t_i} F_n \end{eqnarray}

To correct for that, we can multiply the result with the Nyquist frequency phase $e^{-i\omega_{-N/2} t_i}$:

\begin{eqnarray} f(t_i) = \frac{1}{\sqrt{N}} \sum_{n=-N/2}^{n=N/2-1} e^{-i\omega_n t_i} F_n = e^{-i\omega_{-N/2} t_i} \frac{1}{\sqrt{N}} \sum_{n=-N/2}^{N/2-1} e^{-i(\omega_n-\omega_{-N/2}) t_i} F_n= e^{-i\omega_{-N/2} t_i}\frac{1}{\sqrt{N}} \sum_{n=0}^{N-1} e^{-i\omega_n t_i} F_n \end{eqnarray}

from numpy import *

plt.plot(ts,(ft * exp(-1j*ts*ws[0])).real)

plt.plot(t,y2[:,0])

[<matplotlib.lines.Line2D at 0x165d4de50>]

Now it works. Our plot extends from $[-5,5]$ instead of $[0,10]$, so the values $[5,10]$ are found in $[-5,0]$.

Since this issue with the shift of the function is very common, FFT implements a function called fftshift, which shifts the spectra so that we can start summation with $\omega_n=0$. Check it out below:

Fs2 = fftshift(Fs)
f2=ifft(Fs2)

plt.plot(ws,Fs2,label='shifted' )
plt.plot(ws,Fs,label='original')
plt.legend(loc='best')

<matplotlib.legend.Legend at 0x165c14590>

plt.plot(t,y2[:,0],label='original')
plt.plot(t,f2,label='transformed')

[<matplotlib.lines.Line2D at 0x166118650>]

Now the result is exactly what is expected, even the interval is between $[0,10]$.

Optimization

Optimization (finding minima or maxima of a function) is a large field in mathematics, and optimization of complicated functions or in many variables can be rather involved. Here we will only look at a few very simple cases. For a more detailed introduction to optimization with SciPy see: http://scipy-lectures.github.com/advanced/mathematical_optimization/index.html

To use the optimization module in scipy first include the optimize module:

from scipy import optimize

Finding a minima

Let's first look at how to find the minima of a simple function of a single variable:

def f(x):
    return 4*x**3 + (x-2)**2 + x**4

x = linspace(-5, 3, 100)
plt.plot(x, f(x));