libsvm格式

格式简介

libsvm和libfm格式相同。在推荐系统中由于离散特征过于稀疏，为了减小存储负担，只保存索引值，即值为0的不存储。样例格式如下：

<label> <index1>:<value1> <index2>:<value2> ...
 ...
 ...

通过全数字的方式节省存储空间。其中，index表示第几个特征（事先记录号对应关系），value表示对应值。如果不在乎特征名称所占用的存储，可以直接存储名称，如下：

<label> <feature1>:<value1> <feature2>:<value2> ...
 ...
 ...

该种方式直接存储特征名称和特征值。还可以采用excel形式的csv格式

<label> <feature1> <feature2> ...
<label> <value1> <value2> ...
<label> <value1> <value2> ...
 ...
 ...

csv转libsvm格式

读取每条数据，把有值的特征和值组成键值对，重新写入。

data = pd.read_csv('./data/train_1m.txt', header=None, delimiter='\t',)
data.columns = ['label', 'I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13',
                'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15',
                'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26',
                ]

data = data.fillna('-')

for index, row in data.iterrows():
    txt = ''
    label = str(row[0])
    txt += label
    txt += '\t'
    for i, fea in enumerate(row[1:]):
        if fea != '-':
            txt += data.columns[i + 1]
            txt += ':'
            txt += str(fea)
            if i != len(row[1:]) - 1:
                txt += '\t'
    print(txt)
    with open('./data/train_libfm','a') as f:
        f.write(txt)

libsvm转tf.record格式

tf.record格式是tensorflow的推荐格式。

def float_feature(value):
    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))


def int64_feature(value):
     return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))


def byte_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value.encode()]))


def line_processor(line):
    features = dict()
    line = line.strip().split('\t')
    label = line[0]
    features['label'] = float_feature(float(label))
    for i in range(1, 14):
        features['I' + str(i)] = float_feature(0)
    for i in range(1, 27):
        features['C' + str(i)] = byte_feature('0')

    sparse_features = ['C' + str(i) for i in range(1, 27)]  # 离散特征
    dense_features = ['I' + str(i) for i in range(1, 14)]  # 连续特征

    for fea in line[1:]:
        fea_name, fea_value = fea.split(':')
        if fea_name in sparse_features:
            features[fea_name] = byte_feature(fea_value)
        elif fea_name in dense_features:
            features[fea_name] = float_feature(float(label))

    tf_example = tf.train.Example(features=tf.train.Features(
        feature=features
    ))

    tf_serialized = tf_example.SerializeToString()
    return tf_serialized