First-half of the Semester

It has been half a semester! As a PhD student under Alibaba-NTU Talent Program, for now, my research direction is gravitating towards medical imaging. These 8 weeks have been hectic with course works and admin from both NTU and Alibaba side though. I shall update with content very soon.

The course works I am taking now:

CE7453. Numerical Algorithm. Cubic B-Splines Interpolation.
EE7403. Image Processing.
HP7001. Advanced Research Design and Data Analysis.
K6312. Information Mining and Analysis.

See the notes for the courses above here (except K6312).

[No longer Updated]

Alibaba-NTU Talent Program

Hi! This is week 2 of my PhD at NTU as an employee of Alibaba. I will attempt to clear course requirements – and of course learn new stuffs! Here are the related topics that I will review:

1. Numerical Algorithm
2. Statistics
3. Image Processing

It seems like I will proceed with image processing for my research; but anyway, there are still things pending approvals and what-not.

Update soon!

(20190123) Here is the link to extended summary to Andrew Ng’s coursera machine learning course.

Neural Network Tutorials with keras: a Detour

Hello!

I had planned to develop kero further. But at the end of the day, that should be just for practice; look at the existing libraries and APIs such as keras and tensorflow: they are already very extensive and also well maintained. On that note, I should remind myself that one does not try to beat the giants in their game, unless I have something really innovative to offer.

That said, while implementing kero, I had had fun reading up the theories down to the fundamental of basic implementation of a neural network. For now, I think exploring tutorials with keras is a good idea as well!

To start this off, here is the first tutorial. Enjoy!

Update :
(20190104) second tutorial on MNIST.

Utilities

Example usage: Create “tryy.csv” file containing rows and columns of decimal numbers. Run the following.

import csv
import numpy as np

# filename : string, name of csv file without extension
# headerskip : nonnegative integer, skip the first few rows
#   if all rows are to be read, set to zero

out = []
with open(str(filename)+'.csv') as csv_file:
count = 0
i = 0
count = count + 1
else:
out.append(row)
if get_the_first_N_rows>0:
i = i + 1
if i == get_the_first_N_rows:
break
return out

# example
xx = [[np.float(x) for x in y] for y in xx] # nice here
print(xx)
print(np.sum(xx[0]))

Side Quest: parameter estimate for differential equations

Hi all,

I got a little side quest from a friend doing PhD in NTU. To put it simply, there is a cubic differential equation whose parameters are to be fit with existing data. Since I was playing around with gradient descent recently, I decide to give it a try. You can view it in this link Parameter Estimation for Differential Equations using Gradient Descent including a little more technical note. For that I also write a very simple Runge-Kutta wrapper in this link.

Update: I tested kero 0.6.3 Neural Network library on MNIST here; more work needs to be done!

‘Til next time!

This slideshow requires JavaScript.

Runge-Kutta with wrapper

See the codes for each example at the end of the post.

Example 1

Let us solve the differential equations $\frac{dx}{dt}=-x$. The following wrapper uses Runge-Kutta solver from scipy. The purpose of the wrapper is to compute the specified points $\{(x_i,t_i)\}$.

1. Input time_checkpoints into RK45_wrapper. time_checkpoints is the list [t1,t2,…,tN] whose values of x we want to compute.
2. x0 is the initial value, i.e. the value of x at t1. This is the initial condition for the initial value problem.
3. stepsize. This sets the maximum step size between tk and t(k+1). For example, if tk=1 and t(k+1)=2, stepsize can be set to 0.1 and the values will be computed at 1.1, 1.2, …, 2.0. By default stepsize = None.  When set to None, the stepsize is a-hundreth of tk and t(k+1) interval.

As seen in the figure below, the green “check points” are the values that we specify for the wrapper to compute via time_checkpoints. Note that t_checkpoint_new will be computed, since the solver DOES NOT exactly computes the values at t_checkpoint. However, by setting the stepsize as small as possible, t_checkpoint_new will be closer to t_checkpoint. We print max checkpt diff to show the difference between t_checkpoint_new and t_checkpoint. In this example, using the default stepsize is sufficiently close, i.e. max checkpt diff=0.

Red circles are the solutions generated by RK45 solver, and blue curve is the solution plotted analytically.

Example 2

Now we try out similarly with pendulum equation example like the example here. This shows that essentially the same method works for multi-valued function as well. The equation is given by

$\ddot{\theta}(t)+ b \cdot \dot{\theta}(t) +c\cdot sin(\theta (t)) = 0$

which can be linearized as a system of equations:

$\dot{\theta}(t) = \omega(t)$
$\dot{\omega}(t) = -b\cdot \omega (t) - c\cdot sin(\theta (t))$

We get the following. Each color correspond to one component in the linearized system. The scatter dots are solutions solved by RK45, the cross and circle are respectively the checkpoints we want to compute and the same checkpoints produced by RK45.

Code for Example 1

import numpy as np
from scipy.integrate import RK45
import matplotlib.pyplot as plt

def f(t,y):
# dy/dt = f(t,y)
return np.array([-1*y[0]])

x0 = np.array([1.0]) # initial value
# t0 = 0  # initial time
# t1 = 10 # final time

def RK45_wrapper(x0, time_checkpoints, f, stepsize=None):

if stepsize is None:
stepsize = 0.01*(time_checkpoints[-1]-time_checkpoints[0])
t_set = [time_checkpoints[0]]
x_set = [x0]
t_checkpoint_new = [time_checkpoints[0]]
x_checkpoint = [x0]

for i in range(len(time_checkpoints)-1):
t1 = time_checkpoints[i+1]
if i>0:
x0 = ox.y
t0 = ox.t

else:
t0 = time_checkpoints[i]
ox = RK45(f,t0,x0,t1,max_step=stepsize)

while ox.t < t1:
msg = ox.step()
t_set.append(ox.t)
x_set.append(ox.y)
x_checkpoint.append(ox.y)
t_checkpoint_new.append(ox.t)
return x_set, t_set, x_checkpoint, t_checkpoint_new

# all data by DE (differential equation)
time_checkpoints = np.linspace(0,10,11)
x_set, t_set, x_checkpoint, t_checkpoint_new = RK45_wrapper(x0, time_checkpoints, f, stepsize=None)

t_checkpt_diff = []
for t,t_new in zip(time_checkpoints, t_checkpoint_new):
t_checkpt_diff.append(abs(t-t_new))
print("max checkpt diff = ", np.max(t_checkpt_diff))

# all data, analytic
t0 = time_checkpoints[0]
t1 = time_checkpoints[-1]
t_set2 = np.linspace(t0,t1,100)
f_int = lambda t: np.exp(-t)
x_set2 = [f_int(t_set2[i]) for i in range(len(t_set2))]

fig = plt.figure()
ax.scatter(t_set,x_set,facecolors='none',edgecolors='r',label="all data by DE")
ax.scatter(t_checkpoint_new,x_checkpoint,20,'g',label="check points")
ax.plot(t_set2,x_set2,label="all data, analytic")
ax.set_xlabel("t")
ax.set_ylabel("x")
ax.legend()
plt.show()

Code for Example 2

import numpy as np
from scipy.integrate import RK45
import matplotlib.pyplot as plt

def f(t,y, f_args):
theta, omega = y
dydt = [omega, -f_args[0]*omega - f_args[1]*np.sin(theta)]
return dydt

def RK45_wrapper(x0, time_checkpoints, f, f_args, stepsize=None):
if stepsize is None:
stepsize = 0.01*(time_checkpoints[-1]-time_checkpoints[0])
t_set = [time_checkpoints[0]]
x_set = [x0]
t_checkpoint_new = [time_checkpoints[0]]
x_checkpoint = [x0]

f_temp = lambda t,y : f(t,y, f_args)

for i in range(len(time_checkpoints)-1):
t1 = time_checkpoints[i+1]
if i>0:
x0 = ox.y
t0 = ox.t

else:
t0 = time_checkpoints[i]
ox = RK45(f_temp,t0,x0,t1,max_step=stepsize)

while ox.t < t1:
msg = ox.step()
t_set.append(ox.t)
x_set.append(ox.y)
x_checkpoint.append(ox.y)
t_checkpoint_new.append(ox.t)
return x_set, t_set, x_checkpoint, t_checkpoint_new

b = 0.25
c = 5.0
f_args = [b,c]
x0 = [np.pi - 0.1, 0.0]
time_checkpoints = np.linspace(0, 10, 11)
x_set, t_set, x_checkpoint, t_checkpoint_new = RK45_wrapper(x0, time_checkpoints, f, f_args, stepsize=None)

# print(np.array(x_set).shape) # (112, 2)
fig=plt.figure()
ax.scatter(t_set,[x[0] for x in x_set],3,label="RK45 x[0]")
ax.scatter(t_set,[x[1] for x in x_set],3,label="RK45 x[1]")
ax.scatter(t_checkpoint_new,[x[0] for x in x_checkpoint],50,label="output ckpt x[1]",facecolors='none',edgecolors='b')
ax.scatter(t_checkpoint_new,[x[1] for x in x_checkpoint],50,label="output ckpt x[1]",facecolors='none',edgecolors='r')
ax.scatter(time_checkpoints,[x[0] for x in x_checkpoint],30,label="manual ckpt x[1]",marker="x",color="b")
ax.scatter(time_checkpoints,[x[1] for x in x_checkpoint],30,label="manual ckpt x[1]",marker="x",color="r")

ax.set_xlabel("x")
ax.set_ylabel("y")
ax.legend()
plt.show()

print_numpy_matrix()

Print numpy matrix with nice indentation.

kero.utils.utils.py

def print_numpy_matrix(np_mat,formatting="%6.2f",no_of_space=20):
return

Arguments/Return

 np_mat,formatting String. The usual format for printing in python. Default=”%6.2f”S no_of_space Integer. Number of spaces from the left with which the matrix is indented. Default=20

Example Usage 1

import kero.utils.utils as ut
import numpy as np

x=np.matrix(np.random.normal(size=(3,4)))
print("x = ")
print(x)
print("\nwith print_numpy_matrix(): ")
print("x = ")
ut.print_numpy_matrix(x,formatting="%6.5f",no_of_space=10)

whose output is

x =
[[ 1.61318393 -0.37102784 -0.89363802 -0.56962807]
[-0.08657515 1.20537268 -1.95176244 -1.84425267]
[-0.59647232 -0.96593909 1.02197755 0.39493434]]

with print_numpy_matrix():
x =
1.61318 -0.37103 -0.89364 -0.56963
-0.08658 1.20537 -1.95176 -1.84425
-0.59647 -0.96594 1.02198 0.39493

Example Usage 2

kero version: 0.6.2

numpy_matrix_to_list()

Convert a numpy matrix to a list.

kero.utils.utils.py

def numpy_matrix_to_list(np_mat):
return out

Arguments/Return

 np_mat Numpy matrix. Return out List.

Example Usage 1

testUtils3.py

import kero.utils.utils as ut
import numpy as np

n_row = 3
n_col = 2
M = np.random.normal(size=(n_row,n_col))
print(M)
print("------convert to list-----")
out = ut.numpy_matrix_to_list(M)
print(out)
print("------convert back to matrix--------")
MM = np.matrix(out)
print(MM)

The output is the following.

[[-0.07988029  1.2434192 ]
[-0.5162123  -0.59034063]
[-1.02402997 -0.74755916]]
------convert to list-----
[[-0.07988028825737249, 1.24341920016077], [-0.5162122966577327, -0.5903406251819941], [-1.024029973127149, -0.7475591629901397]]
------convert back to matrix--------
[[-0.07988029  1.2434192 ]
[-0.5162123  -0.59034063]
[-1.02402997 -0.74755916]]

kero version: 0.6.2

partition_list()

Partition a list into a variety of partitions. For example, [1,2,3,4] can be partitioned into [[1,2],[3,4]].

kero.utils.utils.py

def partition_list(x, partition, do_shuffle=False,
verbose=False,
for_index=False):
return collection_of_batches , index_partition

Arguments/Return

 x list partition Either integer or list of list. Integer. If integer>0 is given, then each batch size is the size of the integer. For example partition=3 to list=[1,2,3,4,5,6,7] will yield [1,2,3],[4,5,6],[7,8]]. List of list. [ind1, ind2, …, indN] where ind is a list of indices    for example, partition = [[2,1],[3]] for list=[x,y,z] will yield [[y,x],[z]]. do_shuffle Bool. If true, shuffle the content of the partition result. Default=False verbose Bool False or integer The larger the integer, the more information is printed. Set them to suitable integers for debugging. Default=False for_index Bool. (for internal recursion, not meant for usage) Default=False Return collection_of_batches List of list [batch], where each batch is a sub-list of x Return index_partition List of list. If not None, then this is the list of indices of x, corresponding to the output. For example if x=[a,b,c,d] and the collection_of_batches is [[b,a],[d,c]], then this index_partition is [[1,0],[3,2]]

Example Usage 1

testUtils2.py

import kero.utils.utils as ut
import numpy as np

N=24

x1 = range(N)
x2 = ["a","b","c","d","e","f","g","h","i","j","k"]
batch_sizes = [7,4]

for x,batch_size in zip([x1,x2],batch_sizes ):
for shuffle in [False,True]:
print("size | items | indices (shuffle =", shuffle,")")
print("---------------------------------")
collection_of_batches, index_partition = ut.partition_list(x,batch_size, do_shuffle=shuffle)
total_size = 0
for y,ind in zip(collection_of_batches,index_partition):
print(" ",len(y)," : ",y , " : ", ind)
total_size = total_size + len(y)
print("---------------------------------")
print("total size = ", total_size,"\n")

print("----------using manual index---------------")
manual_index = [[1,2,3,4,5],[6,7],[0,10,9,8]]
partition=manual_index
collection_of_batches, _ = ut.partition_list(x,partition )
print("size | items | indices (shuffle =", shuffle,")")
print("---------------------------------")
for y in collection_of_batches:
print(" ",len(y)," : ",y)

The output is the following.

size | items | indices (shuffle = False )
---------------------------------
7 : [0, 1, 2, 3, 4, 5, 6] : [0, 1, 2, 3, 4, 5, 6]
7 : [7, 8, 9, 10, 11, 12, 13] : [7, 8, 9, 10, 11, 12, 13]
7 : [14, 15, 16, 17, 18, 19, 20] : [14, 15, 16, 17, 18, 19, 20]
3 : [21, 22, 23] : [21, 22, 23]
---------------------------------
total size = 24

size | items | indices (shuffle = True )
---------------------------------
7 : [13, 6, 10, 4, 22, 11, 9] : [13, 6, 10, 4, 22, 11, 9]
7 : [21, 20, 14, 8, 0, 23, 17] : [21, 20, 14, 8, 0, 23, 17]
7 : [15, 5, 3, 19, 16, 18, 12] : [15, 5, 3, 19, 16, 18, 12]
3 : [1, 7, 2] : [1, 7, 2]
---------------------------------
total size = 24

size | items | indices (shuffle = False )
---------------------------------
4 : ['a', 'b', 'c', 'd'] : [0, 1, 2, 3]
4 : ['e', 'f', 'g', 'h'] : [4, 5, 6, 7]
3 : ['i', 'j', 'k'] : [8, 9, 10]
---------------------------------
total size = 11

size | items | indices (shuffle = True )
---------------------------------
4 : ['f', 'j', 'e', 'b'] : [5, 9, 4, 1]
4 : ['c', 'g', 'd', 'a'] : [2, 6, 3, 0]
3 : ['i', 'h', 'k'] : [8, 7, 10]
---------------------------------
total size = 11

----------using manual index---------------
size | items | indices (shuffle = True )
---------------------------------
5 : ['b', 'c', 'd', 'e', 'f']
2 : ['g', 'h']
4 : ['a', 'k', 'j', 'i']

kero version: 0.6.2