partition_list()

home > kero > Documentation

Partition a list into a variety of partitions. For example, [1,2,3,4] can be partitioned into [[1,2],[3,4]].

kero.utils.utils.py

def partition_list(x, partition, do_shuffle=False,
	verbose=False,
	for_index=False):
  return collection_of_batches , index_partition

Arguments/Return

x list
partition Either integer or list of list.

Integer. If integer>0 is given, then each batch size is the size of the integer. For example partition=3 to list=[1,2,3,4,5,6,7] will yield [1,2,3],[4,5,6],[7,8]].

List of list. [ind1, ind2, …, indN] where ind is a list of indices    for example, partition = [[2,1],[3]] for list=[x,y,z] will yield [[y,x],[z]].

do_shuffle Bool. If true, shuffle the content of the partition result.

Default=False

verbose Bool False or integer

The larger the integer, the more information is printed. Set them to suitable integers for debugging.

Default=False

for_index Bool. (for internal recursion, not meant for usage)

Default=False

Return collection_of_batches List of list [batch], where each batch is a sub-list of x
Return

index_partition

List of list. If not None, then this is the list of indices of x, corresponding to the output. For example if x=[a,b,c,d] and the collection_of_batches is [[b,a],[d,c]], then this index_partition is [[1,0],[3,2]]

Example Usage 1

testUtils2.py

import kero.utils.utils as ut
import numpy as np

N=24

x1 = range(N)
x2 = ["a","b","c","d","e","f","g","h","i","j","k"]
batch_sizes = [7,4]


for x,batch_size in zip([x1,x2],batch_sizes ):
	for shuffle in [False,True]:
		print("size | items | indices (shuffle =", shuffle,")")
		print("---------------------------------")
		collection_of_batches, index_partition = ut.partition_list(x,batch_size, do_shuffle=shuffle)
		total_size = 0
		for y,ind in zip(collection_of_batches,index_partition):
			print(" ",len(y)," : ",y , " : ", ind)
			total_size = total_size + len(y)
		print("---------------------------------")
		print("total size = ", total_size,"\n")

print("----------using manual index---------------")
manual_index = [[1,2,3,4,5],[6,7],[0,10,9,8]]
partition=manual_index
collection_of_batches, _ = ut.partition_list(x,partition )
print("size | items | indices (shuffle =", shuffle,")")
print("---------------------------------")
for y in collection_of_batches:
	print(" ",len(y)," : ",y)

The output is the following.

size | items | indices (shuffle = False )
---------------------------------
7 : [0, 1, 2, 3, 4, 5, 6] : [0, 1, 2, 3, 4, 5, 6]
7 : [7, 8, 9, 10, 11, 12, 13] : [7, 8, 9, 10, 11, 12, 13]
7 : [14, 15, 16, 17, 18, 19, 20] : [14, 15, 16, 17, 18, 19, 20]
3 : [21, 22, 23] : [21, 22, 23]
---------------------------------
total size = 24

size | items | indices (shuffle = True )
---------------------------------
7 : [13, 6, 10, 4, 22, 11, 9] : [13, 6, 10, 4, 22, 11, 9]
7 : [21, 20, 14, 8, 0, 23, 17] : [21, 20, 14, 8, 0, 23, 17]
7 : [15, 5, 3, 19, 16, 18, 12] : [15, 5, 3, 19, 16, 18, 12]
3 : [1, 7, 2] : [1, 7, 2]
---------------------------------
total size = 24

size | items | indices (shuffle = False )
---------------------------------
4 : ['a', 'b', 'c', 'd'] : [0, 1, 2, 3]
4 : ['e', 'f', 'g', 'h'] : [4, 5, 6, 7]
3 : ['i', 'j', 'k'] : [8, 9, 10]
---------------------------------
total size = 11

size | items | indices (shuffle = True )
---------------------------------
4 : ['f', 'j', 'e', 'b'] : [5, 9, 4, 1]
4 : ['c', 'g', 'd', 'a'] : [2, 6, 3, 0]
3 : ['i', 'h', 'k'] : [8, 7, 10]
---------------------------------
total size = 11

----------using manual index---------------
size | items | indices (shuffle = True )
---------------------------------
5 : ['b', 'c', 'd', 'e', 'f']
2 : ['g', 'h']
4 : ['a', 'k', 'j', 'i']

kero version: 0.6.2