MNIST Neural Network test 1

home > Machine Learning

To test MNIST using kero 0.6.3, I will use jupyter notebook in a virtual environment. Also, in this folder, place adhoc_utils.py containing the function read_csv() from here. I will use virtualenv as usual: see here. Then after activating the virtual environment, simply:

pip install jupyter
pip install kero
pip install matplotlib
pip install opencv-python
jupyter notebook

Download MNIST file that has been converted into CSV form; I got it from this link. Now, create the python notebook  mnist_dnn.ipynb (see below) and run all the cellsYou can find this test run and similar test runs here.

Unfortunately, it appears that the trained models only predict one single output for any input (it predicts only 6 for any image in one of the attempts, which is bad). Several possible issues and remarks include the following.

  1. There might be defective data points. Update: not likely, it is easy to check it with tested machine learning algorithm. I tried using keras on the same data here; training and prediction have been successful.
  2. Different loss functions are more suitable, check out, for example, KL divergence. Update: this is certainly more than meets the eye. See a tutorial from Stanford here. Using MSE, which is L2, appears to be harder to optimize. Use instead L1 norms like cross-entropy loss.
  3. This example uses no softmax layer at the end; in fact, using default Neural Network from kero, the final layer is activated using the same activation function (in this example, sigmoid function) as other layers. The maximum value at the output layer is taken as the predicted output.
  4. DNN has been treated like a black box; nobody quite knows what happens throughout the process in a coherent manner; in fact it could be just that the randomly initialized weights before training were not chosen in the right range. This might be interesting to study in the future (hopefully the experts come out with new insights soon).

All the above said, the little modification I did (before softmax) includes initiating all biases to zero instead of random and allow for options to generate random weights in a normalized manner (that depend on the number of neurons). I might change the interface a little, but in any case, seems like there might be more works to do! That’s all for now, happy new year!

mnist_dnn.ipynb

[1]

import numpy as np
import adhoc_utils as Aut
import matplotlib.pyplot as plt
import cv2, time

import kero.multib.NeuralNetwork as nn
import kero.utils.utils as ut

[2]

# Loading MNIST image data from csv files. 
# Also, binarize labels.
#
#

def simple_binarizer(mnist_label, bin_factor=1, bin_shift=0):
    # mnist_label: int, 0,1,2... or 9
    out = [0+bin_shift]*10
    out[mnist_label] = 1*bin_factor
    return np.transpose(np.matrix(out))
def convert_list_of_string_to_float(this_list):
    out = []
    for i in range(len(this_list)):
        out.append(float(this_list[i]))
    return out

bin_shift = 0
bin_factor = 1
img_width, img_height = 28, 28
pixel_normalizing_factor = 255
# read_csv returns list of list.
# good news is, the loaded data is already flattened.
mnist_train =  Aut.read_csv("mnist_train", header_skip=1,get_the_first_N_rows = 6400)
mnist_train_labels_binarized = [simple_binarizer(int(x[0]),bin_factor=bin_factor,bin_shift=bin_shift) for x in mnist_train]
mnist_train_data = [1/pixel_normalizing_factor*np.transpose(np.matrix(convert_list_of_string_to_float(x[1:]))) for x in mnist_train]
# 

# Uncomment this to print the binarized labels
#
for i in range(5):
    print(mnist_train[i][0] ,":",ut.numpy_matrix_to_list(mnist_train_labels_binarized[i]))

[3]

# Uncomment this to see the flattened image profile
#
# temp = mnist_train_data[0]
# print("max = ", np.max(temp))
# print("min = ", np.min(temp))
# mean_val = np.mean(temp)
# print("mean = ", mean_val)
# fig0 = plt.figure()
# ax0 = fig0.add_subplot(111)
# ax0.plot(range(len(temp)),temp)
# ax0.plot(range(len(temp)),[mean_val]*len(temp))

[4]

# To visualize the loaded data, uncomment and run this section.
#
# 

# mnist_train_labels = [x[0] for x in mnist_train]
# mnist_train_data_image_form = [np.array(x[1:]).reshape(img_height,img_width).astype(np.uint8) for x in mnist_train]

# data_length = len(mnist_train_data)
# for i in range(10):
#     if i < data_length:
#         print(mnist_train_data_image_form[i].shape,end=",")

# #  
# count=0
# title_set = []
# for label,img_data in zip(mnist_train_labels,mnist_train_data_image_form):
#     title = "count: "+str(count)+"| label: "+str(label)
#     title_set.append(title)
#     cv2.imshow(title, img_data)
#     cv2.resizeWindow(title, 300,300)
#     count = count + 1
#     if count == 5:
#         break
# cv2.waitKey(0)
# for title in title_set:
#     cv2.destroyWindow(title)

[5]

# input_set: list of numpy matrix [x], 
#   where each x is a column vector m by 1, m the size of input layer.
# Y0_set: list of numpy matrix [Y0],
#   where each Y0 is a column vector N by 1, N the size of output layer.
#   This is equal to 10, since it corresponds to labels 0,1,...,9.
#
#

input_set = mnist_train_data
Y0_set = mnist_train_labels_binarized
number_of_neurons = [784,28,10]
lower_bound, upper_bound = 0 ,1
bounds = [lower_bound, upper_bound]
bulk = {
    "number_of_neurons" : number_of_neurons,
    "bounds": bounds,
    "layerwise_normalization": True,
}

NeuralNetwork = nn.NeuralNetwork()
NeuralNetwork.learning_rate = 1
NeuralNetwork.initiate_neural_network(bulk, mode="UniformRandom",
    verbose = False,
    verbose_init_mode=False,
    verbose_consistency=False)

nu = nn.NetworkUpdater()
nu.set_settings(method="RegularStochastic",
method_specific_settings={
        "batch_size":8,
        "no_of_epoch":32,
        "shuffle_batch":True,
})
nu.set_training_data(input_set,Y0_set)
nu.set_neural_network(NeuralNetwork)

[6]

AF = nn.activationFunction(func = "Sigmoid")
start = time.time()
weights_next, biases_next, mse_list = nu.update_wb(input_set, Y0_set, 
                NeuralNetwork.weights, NeuralNetwork.biases, AF,
                mse_mode="compute_and_print", verbose=11)
end = time.time()
elapsed = end - start

[7]

print("epoch | mse value ")
mark = 1
for i in range(len(mse_list)):
    if mark >= 0.1*len(mse_list) or i==0:
        print(" + epoch {",i ,"} ", mse_list[i])
        mark = 1
    else:
        mark = mark + 1

fig = plt.figure()
ax1 = fig.add_subplot(211)
plt.plot(range(len(mse_list)), mse_list)

[8]

print("time taken [s] = ", elapsed)
print("time taken [min] = ", elapsed/60)
print("time taken [hr] = ", elapsed/3600)
print("time taken at 10k x [s] = ", elapsed*1e4)
print("time taken at 10k x [min] = ", elapsed*1e4/(60))
print("time taken at 10k x [hr] = ", elapsed*1e4/(3600))
print("time taken at 10k x [day] = ", elapsed*1e4/(3600*24))

[9]

no_of_images_to_test=500
mnist_test =  Aut.read_csv("mnist_test", header_skip=1,get_the_first_N_rows = no_of_images_to_test)
mnist_test_labels = [int(x[0]) for x in mnist_test]
mnist_test_data = [1/pixel_normalizing_factor*np.transpose(np.matrix(convert_list_of_string_to_float(x[1:]))) for x in mnist_test]

hit_list = []
predict_list = []
predict_val = []
for i in range(no_of_images_to_test):
    a_1 = mnist_test_data[i]
    test_a_l_set, _ = nu.feed_forward(weights_next, biases_next, a_1, AF,
        verbose=False,
        matrix_formatting="%6.2f")
    Y_after = test_a_l_set[-1]
    predicted_label = int(np.argmax(Y_after))
    actual_label= mnist_test_labels[i]
    # print(Y_after)
#     print("predicted vs actual = ", predicted_label,"/",actual_label)
    predict_list.append(predicted_label)
    predict_val.append(Y_after)
    if actual_label==predicted_label:
        hit_list.append(1)
    else:
        hit_list.append(0)
print("predict list = ")
print(predict_list)
print("predict values = ")
for i in range(10):
#     print(ut.numpy_matrix_to_list(predict_val[i]))
    ut.print_numpy_matrix(np.transpose(predict_val[i]),formatting="%9.6f",no_of_space=20)
print("hit list = ")
print(hit_list)
print("percentage correct = ", 100* np.sum(hit_list)/len(hit_list))

 

kero version 0.6.3

print_numpy_matrix()

home > kero > Documentation

Print numpy matrix with nice indentation.

kero.utils.utils.py

def print_numpy_matrix(np_mat,formatting="%6.2f",no_of_space=20):
  return

Arguments/Return

np_mat,formatting String. The usual format for printing in python.

Default=”%6.2f”S

no_of_space Integer. Number of spaces from the left with which the matrix is indented.

Default=20

Example Usage 1

import kero.utils.utils as ut
import numpy as np

x=np.matrix(np.random.normal(size=(3,4)))
print("x = ")
print(x)
print("\nwith print_numpy_matrix(): ")
print("x = ")
ut.print_numpy_matrix(x,formatting="%6.5f",no_of_space=10)

whose output is

x =
[[ 1.61318393 -0.37102784 -0.89363802 -0.56962807]
[-0.08657515 1.20537268 -1.95176244 -1.84425267]
[-0.59647232 -0.96593909 1.02197755 0.39493434]]

with print_numpy_matrix():
x =
1.61318 -0.37103 -0.89364 -0.56963
-0.08658 1.20537 -1.95176 -1.84425
-0.59647 -0.96594 1.02198 0.39493

Example Usage 2

See Deep Learning and Neural Network with kero PART 1.

kero version: 0.6.2

numpy_matrix_to_list()

home > kero > Documentation

Convert a numpy matrix to a list.

kero.utils.utils.py

def numpy_matrix_to_list(np_mat):
  return out

Arguments/Return

np_mat Numpy matrix.
Return out List.

Example Usage 1

testUtils3.py

import kero.utils.utils as ut
import numpy as np

n_row = 3
n_col = 2
M = np.random.normal(size=(n_row,n_col))
print(M)
print("------convert to list-----")
out = ut.numpy_matrix_to_list(M)
print(out)
print("------convert back to matrix--------")
MM = np.matrix(out)
print(MM)

The output is the following.

[[-0.07988029  1.2434192 ]
 [-0.5162123  -0.59034063]
 [-1.02402997 -0.74755916]]
------convert to list-----
[[-0.07988028825737249, 1.24341920016077], [-0.5162122966577327, -0.5903406251819941], [-1.024029973127149, -0.7475591629901397]]
------convert back to matrix--------
[[-0.07988029  1.2434192 ]
 [-0.5162123  -0.59034063]
 [-1.02402997 -0.74755916]]

kero version: 0.6.2

partition_list()

home > kero > Documentation

Partition a list into a variety of partitions. For example, [1,2,3,4] can be partitioned into [[1,2],[3,4]].

kero.utils.utils.py

def partition_list(x, partition, do_shuffle=False,
	verbose=False,
	for_index=False):
  return collection_of_batches , index_partition

Arguments/Return

x list
partition Either integer or list of list.

Integer. If integer>0 is given, then each batch size is the size of the integer. For example partition=3 to list=[1,2,3,4,5,6,7] will yield [1,2,3],[4,5,6],[7,8]].

List of list. [ind1, ind2, …, indN] where ind is a list of indices    for example, partition = [[2,1],[3]] for list=[x,y,z] will yield [[y,x],[z]].

do_shuffle Bool. If true, shuffle the content of the partition result.

Default=False

verbose Bool False or integer

The larger the integer, the more information is printed. Set them to suitable integers for debugging.

Default=False

for_index Bool. (for internal recursion, not meant for usage)

Default=False

Return collection_of_batches List of list [batch], where each batch is a sub-list of x
Return

index_partition

List of list. If not None, then this is the list of indices of x, corresponding to the output. For example if x=[a,b,c,d] and the collection_of_batches is [[b,a],[d,c]], then this index_partition is [[1,0],[3,2]]

Example Usage 1

testUtils2.py

import kero.utils.utils as ut
import numpy as np

N=24

x1 = range(N)
x2 = ["a","b","c","d","e","f","g","h","i","j","k"]
batch_sizes = [7,4]


for x,batch_size in zip([x1,x2],batch_sizes ):
	for shuffle in [False,True]:
		print("size | items | indices (shuffle =", shuffle,")")
		print("---------------------------------")
		collection_of_batches, index_partition = ut.partition_list(x,batch_size, do_shuffle=shuffle)
		total_size = 0
		for y,ind in zip(collection_of_batches,index_partition):
			print(" ",len(y)," : ",y , " : ", ind)
			total_size = total_size + len(y)
		print("---------------------------------")
		print("total size = ", total_size,"\n")

print("----------using manual index---------------")
manual_index = [[1,2,3,4,5],[6,7],[0,10,9,8]]
partition=manual_index
collection_of_batches, _ = ut.partition_list(x,partition )
print("size | items | indices (shuffle =", shuffle,")")
print("---------------------------------")
for y in collection_of_batches:
	print(" ",len(y)," : ",y)

The output is the following.

size | items | indices (shuffle = False )
---------------------------------
7 : [0, 1, 2, 3, 4, 5, 6] : [0, 1, 2, 3, 4, 5, 6]
7 : [7, 8, 9, 10, 11, 12, 13] : [7, 8, 9, 10, 11, 12, 13]
7 : [14, 15, 16, 17, 18, 19, 20] : [14, 15, 16, 17, 18, 19, 20]
3 : [21, 22, 23] : [21, 22, 23]
---------------------------------
total size = 24

size | items | indices (shuffle = True )
---------------------------------
7 : [13, 6, 10, 4, 22, 11, 9] : [13, 6, 10, 4, 22, 11, 9]
7 : [21, 20, 14, 8, 0, 23, 17] : [21, 20, 14, 8, 0, 23, 17]
7 : [15, 5, 3, 19, 16, 18, 12] : [15, 5, 3, 19, 16, 18, 12]
3 : [1, 7, 2] : [1, 7, 2]
---------------------------------
total size = 24

size | items | indices (shuffle = False )
---------------------------------
4 : ['a', 'b', 'c', 'd'] : [0, 1, 2, 3]
4 : ['e', 'f', 'g', 'h'] : [4, 5, 6, 7]
3 : ['i', 'j', 'k'] : [8, 9, 10]
---------------------------------
total size = 11

size | items | indices (shuffle = True )
---------------------------------
4 : ['f', 'j', 'e', 'b'] : [5, 9, 4, 1]
4 : ['c', 'g', 'd', 'a'] : [2, 6, 3, 0]
3 : ['i', 'h', 'k'] : [8, 7, 10]
---------------------------------
total size = 11

----------using manual index---------------
size | items | indices (shuffle = True )
---------------------------------
5 : ['b', 'c', 'd', 'e', 'f']
2 : ['g', 'h']
4 : ['a', 'k', 'j', 'i']

kero version: 0.6.2

update_wb_regular_stochastic()

home > kero > Documentation

Perform regular stochastic mode of gradient descent. See Data Science, Review.

kero.multib.NeuralNetwork.py

class NetworkUpdater:
  def update_wb_regular_stochastic(self, input_set, Y0_set,
		weights, biases, AF,		
		verbose=False,
		verbose_feed_forward=False,
		verbose_compute_diff=False,
		verbose_delta_L=False,
		verbose_compute_delta=False)
    return weights_next, biases_next

Arguments/Return

input_set list of numpy matrix [x]. Each x a column vector m x 1, m the number of neurons in input layer
Y0_set list of numpy matrix [Y0]. Each Y0 nx1, where n is the no of neurons in layer l=L. The true/observed values in the output layer corresponding to the input set. In another words, for each k=1,…,N, Y0_set[k] = f(x[k]) where f is the true function that our neural network is modelling and N the number of data points.
weights The collection of weights in the neural network.

weights is a list [w_l], where w_l is the collection of weights between the (l-1)-th and l-th layer, for l=2,3,…,L where l=1 is the input layer, l=2 the first hidden layer ad and l=L is the output layer.

w_l is a matrix (list of list) so that w_l[i][j] is the weight between neuron j at layer l-1 and neuron i at layer l.

biases the collection of biases in the neural network.

biases is a list [b_l], where b_l is the collection of biases in the l-th layer for l=2,3,…,L

AF AF (activationFunction). Assume it is initiated.
verbose
verbose_feed_forward
verbose_compute_diff
verbose_delta_L
verbose_compute_delta
Bool False or integer

The larger the integer, the more information is printed. Set them to suitable integers for debugging.

 

All default=False

return weights_next Same as weights, but has undergone 1 gradient descent iteration.
return biases_next Same as biases, but has undergone 1 gradient descent iteration.

 

Example Usage 1

Note that in order to use the following script, we need prep() function which is available in Deep Learning and Neural Network with kero PART 1.

testNNupdater2B.py

import testNNupdater2aux as taux
import kero.multib.NeuralNetwork as nn
import kero.utils.utils as ut
import numpy as np
import time

print("--- test2B ---")

# input_set : list of numpy matrix. 
# Y_set : list of numpy matrix. Output computed by NN
# Y0_set : list of numpy matrix. True/observed output
#  the grand objective is to train NN so that Y_set is equal to Y0_set 
# -------------------------------------------
# this is a collection of a_l_set and z_l_set over all data points
#   z_l_set is the collection of z values over all layers, l=2,3,...L
#   and a_l_set is the corresponding activated values
#   Recall: a_l_set and z_l_set each is a list of numpy matrices 

out = taux.prep(print_data=False)
input_set=out["input_set"]
Y_set=out["Y_set"]
Y0_set=out["Y0_set"]
collection_of_fed_forward_a_l=out["collection_of_fed_forward_a_l"]
collection_of_fed_forward_z_l=out["collection_of_fed_forward_z_l"]
weights=out["weights"]
biases=out["biases"]
NeuralNetwork=out["NeuralNetwork"]

a_L_set = Y_set
# weights = [w_2,w_3,w_4]
# biases = [b_2,b_3,b_4]


nu = nn.NetworkUpdater()
nu.set_settings(method="RegularStochastic",
		method_specific_settings={
		"batch_size":4,
		"no_of_epoch":1,
		"shuffle_batch":True,
		})
nu.set_training_data(input_set,Y0_set)
nu.set_neural_network(NeuralNetwork) 

L = len(weights) + 1
n = len(input_set)
AF = nn.activationFunction(func = "Sigmoid")
start = time.time()
weights_next, biases_next = nu.update_wb_regular_stochastic(input_set, Y0_set,
		weights, biases, AF,
		verbose=31,
		verbose_feed_forward=False,
		verbose_compute_diff=False,
		verbose_delta_L=False,
		verbose_compute_delta=False)

end = time.time()
elapsed = end - start

print("weights and biases:")
for W, B in zip(weights_next, biases_next):
	print(" > ", np.matrix(W).shape," | ", np.matrix(B).shape)


print("n (no of data points) = ",n)
print("time taken [s] = ", elapsed)
print("time taken at 10 k steps [s] = ", elapsed*1e4)
print("time taken at 10 k steps [min] = ", elapsed*1e4/(60))
print("time taken at 10 k steps [hr] = ", elapsed*1e4/(3600))
print("time taken at 10 k steps [day] = ", elapsed*1e4/(3600*24))

The output is the following. At high verbose level, this function also prints the layer and its corresponding index in the list (for detailed debugging, if you are interested in tweaking the source code). We show that the dimensions of weight matrices and biases are indeed correct, as shown under “weights and biases”. The time taken in seconds, minutes, hours and days are computed should the process is prolonged 10,000 k, just a rough estimate for scaling up the use of this function.

---aux---
Initializing a Neural Network object.
--- test2B ---
Initializing a Neural Network object.
 -+ update_wb_regular_stochastic().
    Layer (Output):  4  || i =  0 / 2
    Layer:  3  || i =  1 / 2
    Layer:  2  || i =  2 / 2
weights and biases:
 >  (3, 3)  |  (3, 1)
 >  (2, 3)  |  (2, 1)
 >  (2, 2)  |  (2, 1)
n (no of data points) =  24
time taken [s] =  0.015958786010742188
time taken at 10 k steps [s] =  159.58786010742188
time taken at 10 k steps [min] =  2.6597976684570312
time taken at 10 k steps [hr] =  0.04432996114095052
time taken at 10 k steps [day] =  0.0018470817142062718

kero version: 0.6.2

compute_delta_l_per_data_point()

home > kero > Documentation

Compute the value of \delta^l shown in Neural Network and Back Propagation.

kero.multib.NeuralNetwork.py

class NetworkUpdater:
  def compute_delta_l_per_data_point(self, w_l_plus_1, delta_l_plus_1, z_l, AF,
      verbose=False,
      print_format="%6.8f"):
    return delta_l

Arguments/Return

w_l_plus_1 numpy matrix. Matrix of size m x n, where m and n are the number of neurons in the (l+1)-th and l-th layers respectively. In the neural network, this is the weights between layer l and layer l+1.
delta_l_plus_1 numpy matrix. delta value from layer l+1. We are back-propagating using this function.
z_l numpy matrix. Vector of size m x 1, where m is the number of neurons in layer l. In the neural network this is the values at layer l before activation function.
AF AF (activationFunction). Assume it is initiated.
verbose Bool False or integer

The larger the integer, the more information is printed. Set them to suitable integers for debugging.

Default=False

print_format String. Format for printing numpy matrices when verbose is beyond some value.

Default=”%6.8f”

return delta_l numpy matrix. Vector of size m x 1 where m is the number of neurons in layer l.

Example Usage 1

See compute_delta_L_per_data_point().

kero version: 0.6.2

compute_delta_L_per_data_point()

home > kero > Documentation

Compute the value of \delta^L shown in Neural Network and Back Propagation.

kero.multib.NeuralNetwork.py

class NetworkUpdater:
  def compute_delta_L_per_data_point(self, z_L, Y0, Y_set, Y0_set, AF,
      verbose=False,
      print_format="%6.8f"):
    return delta_L

Arguments/Return

z_L numpy matrix. Vector of size m x 1, where m is the number of neurons in the output layer. In the neural network this is the values at the output layer before activation function.
Y0 numpy matrix. Observed/true output data. Vector of size m x 1, where m is the number of neurons in the output layer.
Y_set List of numpy matrix, [Y]. Each numpy matrix Y is a column vector. In a neural network, this is the values at the output layer predicted by the neural network.
Y0_set List of numpy matrix, [Y0]. Each numpy matrix Y is a column vector. There should be equal number of Y in Y_set as Y0 in Y0_set.
AF AF (activationFunction). Assume it is initiated.
verbose Bool False or integer

The larger the integer, the more information is printed. Set them to suitable integers for debugging.

Default=False

print_format String. Format for printing numpy matrices when verbose is beyond some value.

Default=”%6.8f”

return delta_L numpy matrix. Vector of size m x 1 where m is the number of neurons in layer L or output layer.

Example Usage 1

Note that in order to use the following script, we need prep() function which is available in Deep Learning and Neural Network with kero PART 1.

testNNupdater2A.py

import testNNupdater2aux as taux
import kero.multib.NeuralNetwork as nn
import kero.utils.utils as ut
import numpy as np

print("--- test2A ---")

# input_set : list of numpy matrix. 
# Y_set : list of numpy matrix. Output computed by NN
# Y0_set : list of numpy matrix. True/observed output
#  the grand objective is to train NN so that Y_set is equal to Y0_set 
# -------------------------------------------
# this is a collection of a_l_set and z_l_sedbt over all data points
#   z_l_set is the collection of z values over all layers, l=2,3,...L
#   and a_l_set is the corresponding activated values
#   Recall: a_l_set and z_l_set each is a list of numpy matrices 

out = taux.prep(print_data=False)
input_set=out["input_set"]
Y_set=out["Y_set"]
Y0_set=out["Y0_set"]
collection_of_fed_forward_a_l=out["collection_of_fed_forward_a_l"]
collection_of_fed_forward_z_l=out["collection_of_fed_forward_z_l"]
weights=out["weights"]
biases=out["biases"]
NeuralNetwork=out["NeuralNetwork"]

a_L_set = Y_set
# weights = [w_2,w_3,w_4]
# biases = [b_2,b_3,b_4]


nu = nn.NetworkUpdater()
AF = nn.activationFunction(func = "Sigmoid")
for i in range(len(input_set)): # for each data point
	z_l_set = collection_of_fed_forward_z_l[i]

	print("data point ", i," : ")
	# print("  z_L:")
	# ut.print_numpy_matrix(z_L,formatting="%6.2f",no_of_space=10)
	# print("  Y0:")
	# ut.print_numpy_matrix(Y0,formatting="%6.2f",no_of_space=10)
	
	# l = L = 4
	z_L = z_l_set[-1 - 0]
	Y0 = Y0_set[i]
	delta_L = nu.compute_delta_L_per_data_point(z_L, Y0, a_L_set, Y0_set, AF,
			verbose=11,
			print_format="%6.8f")

	# l = 3
	delta_l_plus_1 = delta_L
	w_l_plus_1 = weights[- 1 -0] # w_4
	z_l = z_l_set[-1 -1]
	delta_l = nu.compute_delta_l_per_data_point(w_l_plus_1, delta_l_plus_1, z_l, AF,
		verbose=11,
		print_format="%6.8f")

	# l = 2
	delta_l_plus_1 = delta_l
	w_l_plus_1 = weights[- 1 -1] # w_4
	z_l = z_l_set[-1 -2]
	delta_l = nu.compute_delta_l_per_data_point(w_l_plus_1, delta_l_plus_1, z_l, AF,
		verbose=11,
		print_format="%6.8f")
	# print("------------------------------------------")

We have prepared 24 data points using prep(), and the following shows the output for the first 2 data points.

---aux---
Initializing a Neural Network object.
--- test2A ---
Initializing a Neural Network object.
data point  0  :
 -+ compute_delta_L_per_data_point()
    delta_L =
          0.11825302
          0.12284915
 -+ compute_differential_term_at_l().
    np.array(w_l_plus_1).shape =  (2, 2)
    delta_l =
          0.00598236
          0.00598236
 -+ compute_differential_term_at_l().
    np.array(w_l_plus_1).shape =  (2, 3)
    delta_l =
          0.00029088
          0.00025446
          0.00029721
data point  1  :
 -+ compute_delta_L_per_data_point()
    delta_L =
          0.11825325
          0.12284961
 -+ compute_differential_term_at_l().
    np.array(w_l_plus_1).shape =  (2, 2)
    delta_l =
          0.00598275
          0.00598275
 -+ compute_differential_term_at_l().
    np.array(w_l_plus_1).shape =  (2, 3)
    delta_l =
          0.00029146
          0.00025704
          0.00029734
...

kero version: 0.6.3

select_activation_function()

home > kero > Documentation

kero.multib.NeuralNetwork.py

class activationFunction:
  def select_activation_function(self, func):
    return

Arguments/Return

func String.

If func==”Sigmoid”
Property af of activationFunction set to \sigma(x)=\frac{1}{1+e^{-x}}
Property afp of activationFunction set to \sigma^\prime (x) = \sigma(x) \times (1-\sigma(x))

add more variety of activation functions here

For other unlisted selection of func, then the activation function will be automatically set to “Sigmoid”.

Example Usage 1

See deep learning with kero part 1 and part 2.

kero version: 0.6.2 

activationFunction

home > kero > Documentation

This class of object is designed to contain a variety of activation functions.

kero.multib.NeuralNetwork.py

class activationFunction:
  def __init__(self, func = "Sigmoid"):
    self.af
    self.afp
    return
  def select_activation_function(self, func):
    return
Properties Description
af lambda function. The activation function.
afp lambda function. The derivative of activation function.

kero version: 0.6.2 

MSE()

home > kero > Documentation

MSE(X,Y)=\frac{1}{2n}\Sigma_{i=1}^{n} (X_i-Y_i)^2

kero.multib.NeuralNetwork.py

def MSE(Y_set,Y0_set):
  return mse

Arguments/Return

Y_set List of numpy matrix, [Y]. Each numpy matrix Y is a column vector
Y0_set List of numpy matrix, [Y0]. Each numpy matrix Y is a column vector. There should be equal number of Y in Y_set as Y0 in Y0_set
mse Mean squared values computed by the formula above.

Example Usage 1

testMSE.py

Y0_set= [
  [1,2,3],
  [4,5,6]
]

Y_set = [[1.1*x for x in y] for y in Y0_set]
print("Y0_set=",Y0_set)
print("Y_set=",Y_set)

import numpy as np
# convert to list of numpy matrix
Y0_set = [np.transpose(np.matrix(x)) for x in Y0_set]
Y_set = [np.transpose(np.matrix(x)) for x in Y_set]

import kero.multib.NeuralNetwork as nn

mse=nn.MSE(Y_set,Y0_set)
print("MSE test:\n  mse=",mse)

# manual computation
mse_= 0.01*(1**2+2**2+3**2+4**2+5**2+6**2)
mse_=mse_/(2*2)
print("  compare: ",mse_)

import kero.utils.utils as ut
print("nabla MSE test:\n  nabla mse = ")

nabla_mse = nn.nabla_MSE(Y_set,Y0_set)
ut.print_numpy_matrix(nabla_mse,formatting="%6.2f",no_of_space=5)
nabla_mse_ = 1/2* ((Y_set[0]-Y0_set[0])+(Y_set[1]-Y0_set[1]))
print("  compare: ")
ut.print_numpy_matrix(nabla_mse_,formatting="%6.2f",no_of_space=5)

This will print

Y0_set= [[1, 2, 3], [4, 5, 6]]
Y_set= [[1.1, 2.2, 3.3000000000000003], [4.4, 5.5, 6.6000000000000005]]
MSE test:
  mse= 0.22750000000000029
  compare:  0.2275
nabla MSE test:
  nabla mse =
       0.25
       0.35
       0.45
  compare:
       0.25
       0.35
       0.45

kero version: 0.6.2