keras fit_generator

Let us suppose that your data are images. If you have many images you probably won't be able to load all of them in memory and you would like to read from disk in batches.

Keras flow_from _directory is very fast in doing that as it does this in a multi threading way too but it needs all the images to be in different files, according to their class. If we have all the images in the same file and their classes in separated file we could use the generator bellow to load our x,y data.

 

import pandas as pd
import numpy as np
import cv2

#df_train:  data frame with class of every image
#dpath: path of images

classes=list(np.unique(df_train.label)) 
def batch_generator(ids):
    while True:
        for start in range(0, len(ids), batch_size):
            x_batch = []
            y_batch = []
            end = min(start + batch_size, len(ids))
            ids_batch = ids[start:end]
            for id in ids_batch:
                img = cv2.imread(dpath+'train/{}.png'.format(id)) #open cv read as BGR
                #img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) #BGR to RGB
                #img = cv2.resize(img, (224, 224), interpolation = cv2.INTER_CUBIC)
                #img = pre_process(img)
                labelname=df_train.label.loc[df_train.id==id].values
                labelnum=classes.index(labelname)
                x_batch.append(img)
                y_batch.append(labelnum)
            x_batch = np.array(x_batch)
            y_batch = to_categorical(y_batch,10) 
            yield x_batch, y_batch