入侵检测阅读论文（下）——基于BiLSTM的入侵检测系统代码

基于BiLSTM深度学习算法的入侵检测系统代码
本检测 系统是在《A bidirectional LSTM deep learning approach for intrusion detection》基础上实现的
论文介绍链接: A bidirectional LSTM deep learning approach for intrusion detection

BiLSTM（双向长短期记忆网络）

Bidirectional LSTM(BiDLSTM)是在LSTM基础上提出的，用以提高模型在分类问题上的性能。与LSTM不同的是,BiLSTM网络由前向和后向LSTM组成,其中数据可以向前和向后处理。反向处理捕捉了LSTM通常忽略的数据的隐藏特征和模式,可以更好的捕捉双向的语义依赖,对于提高分类的速度以及准确度具有重要意义。其具体结构如下图所示。


Keras 库使用双向层包装器通过将第一个 LSTM 层作为参数来实现BiLSTM。从图 2 中，输出 (y)、前向隐藏层 (→ ht) 和后向隐藏层 (← ht) 计算如下：
$\overrightarrow{h_t}=H(W_{x\overrightarrow{h}}{x}_t+W_{\overrightarrow{h}\overrightarrow{h}}\overrightarrow{h_{t-1}}+b_{\overrightarrow{h}})$(1)
$\overleftarrow{h_t}=H(W_{x\overleftarrow{h}}{x}_t+W_{\overleftarrow{h}\overleftarrow{h}}\overleftarrow{h_{t+1}}+b_{\overleftarrow{h}})$(2)
$y_t=W_{\overrightarrow{h}y}\overrightarrow{h_t}+W_{\overleftarrow{h}y}\overleftarrow{h_t}+b_y$ (3)

数据集

本系统实现使用了NLS-KDD数据集，该数据集是用来评估入侵检测系统的标准数据集之一,是著名的KDD99数据集的修订版本.由四个子数据集组成:KDDTest+、KDDTest-21、KDDTrain+、KDDTrain+_20Percent.其中KDDTest-21和KDDTrain+_20Percent是KDDTrain+和KDDTest+的子集.数据集每条记录包含 43 个特征,如表1所示.其中41个特征指的是流量输入本身,最后两个是标签(正常或攻击)和分数(流量输入本身的严重性).
数据集下载地址： NLS-KDD

该数据集中包括了4种不同类型的攻击:拒绝服务(Denial of service,DoS)、探测(probe)、用户到根(U2R) 和远程到本地(R2L).同时每种攻击又包含了多种子类型,具体如表2所示。

数据集中数据分布情况如表3所示。

数据预处理

数据归一化及标签编码
由上面的数据集描述可知,NLS-KDD数据集中有三个非数值型的特征,分别是协议类型、服务和标志.但是由于本文所使用的深度学习模型BiDLSTM和LSTM只能处理数值型数据,因此在进行模型训练之前必须进行数据预处理:将非数值型数据转化为数值型数据.特征缩放可以确保数据集处于标准化形式,本文使用了sklearn库进行数据归一化，使用 Min–Max 缩放将每个特征的值缩放到 (0, 1) 范围内。
最小-最大特征缩放的表达式如下：
$$z^{{}^{\prime }}=\frac{X-X_{min}}{X_{max}-X_{min}}$$
实现代码：

from sklearn import preprocessing
import numpy as np
# 原始数据X_train
df = pd.read_csv(filename, header=None)
X_train = df.iloc[:, :df.shape[1] - 1]  # pandas中的iloc切片是完全基于位置的索引
labels = df.iloc[:, df.shape[1] - 1:]
# 初始化数据预处理器，本例中为最小最大值缩放
scaler = MinMaxScaler().fit(X_train)
X_train= scaler.transform(X_train)

数据增强
由于数据集中存在样本不平衡问题，本系统使用了SMOTE过采样技术处理数据。
实现代码：

test_data_filename = "/nls-kdd/datasest/KDDTest+.csv"
X_test1,y_test1=load_data(test_data_filename)
smo = SMOTE(random_state=42)
X_test1, y_test1 = smo.fit_resample(X_test1, y_test1)

BiLSTM模型设计

本系统的BiLSTM模型的各层总结如下：

二分类实现代码：

# bidLSTM
model = Sequential()
model.add(Bidirectional(LSTM(n_hidden, return_sequences=True), input_shape=(41,1)))#
model.add(Dropout(0.2))
model.add(Bidirectional(LSTM(32, return_sequences=True)))
model.add(Dropout(0.2))
model.add(Dense(64, activation='sigmoid'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='sigmoid'))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['acc'])
print(model.summary())

多分类实现代码：

# bidLSTM
model = Sequential()
model.add(Bidirectional(LSTM(n_hidden, return_sequences=True), input_shape=(41,1)))#
model.add(Dropout(0.2))
model.add(Bidirectional(LSTM(32, return_sequences=True)))
model.add(Dropout(0.2))
model.add(Dense(64, activation='sigmoid'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='sigmoid'))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(5, activation='softmax')) #n_classes，输出维度
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])

二分类系统完整代码

二分类检测的完整代码实现如下所示：

import numpy as np
from numpy import float32,int32
np.random.seed(42)
import tensorflow as tf
from keras.layers import TimeDistributed
from keras.layers import Bidirectional

tf.set_random_seed(42)
#线程等设置
session_conf = tf.ConfigProto(
    intra_op_parallelism_threads=1,
    inter_op_parallelism_threads=1
)
from keras import backend as K
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
K.set_session(sess)
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense, Dropout,Flatten
#from data import load_data
from sklearn.preprocessing import Normalizer
from utils import confusion_matrix
from keras.callbacks import TensorBoard, ModelCheckpoint
import os
import matplotlib.pyplot as plt
from sklearn import metrics
from sklearn.preprocessing import OneHotEncoder
import pandas as pd
from sklearn import preprocessing
from sklearn.preprocessing import MinMaxScaler
from keras.utils import to_categorical
from sklearn.metrics import accuracy_score,f1_score,precision_score,recall_score
from sklearn.model_selection import train_test_split
from imblearn.over_sampling import SMOTE
#binary
LABELS = ['normal','abnormal']
lables=np.array(LABELS)
print(lables.shape)
print(lables)
CHECK_ROOT = '/nls-kdd/checkpoint/'
if not os.path.exists(CHECK_ROOT):
    os.makedirs(CHECK_ROOT)
epochs =1 # 30轮次
batch_size = 128#每批次数据量
n_hidden = 64#隐藏单元个数

#加载数据
def load_data(filename):
    df = pd.read_csv(filename, header=None)
    last_column_index = df.shape[1] - 1 
    #标签编码
    df[last_column_index], attacks = pd.factorize(df[last_column_index], sort=True)
    features = df.iloc[:, :df.shape[1] - 1]  # pandas中的iloc切片是完全基于位置的索引
    labels = df.iloc[:, df.shape[1] - 1:]
    return features,labels
raw_data_filename = "/nls-kdd/datasest/KDDTrain+.csv"#换成自己的数据集路径

print("Loading raw data...")
X_train,y_train=load_data(raw_data_filename)
#数据归一化
scaler = MinMaxScaler().fit(X_train)
X_train= scaler.transform(X_train)
np.set_printoptions(precision=3)
y_train = np.array(y_train)

#测试集和验证集划分
X_train, X_test, y_train, y_test = train_test_split(X_train, y_train, train_size=0.8, test_size=0.2, stratify=y_train)
X_train=np.array(X_train)
X_train=np.reshape(X_train,(X_train.shape[0],X_train.shape[1],1))


X_test=np.array(X_test)
X_test=np.reshape(X_test,(X_test.shape[0],X_test.shape[1],1))

# bidLSTM
model = Sequential()
model.add(Bidirectional(LSTM(n_hidden, return_sequences=True), input_shape=(41,1)))#
model.add(Dropout(0.2))
model.add(Bidirectional(LSTM(32, return_sequences=True)))
model.add(Dropout(0.2))
model.add(Dense(64, activation='sigmoid'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='sigmoid'))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['acc'])
print(model.summary())
# callback: draw curve on TensorBoard
tensorboard = TensorBoard(log_dir='/log', histogram_freq=0, write_graph=True, write_images=True)
# callback: save the weight with the highest validation accuracy
filepath=os.path.join(CHECK_ROOT, 'weights-improvement-{val_acc:.4f}-{epoch:04d}.hdf5')
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=2, save_best_only=True, mode='max')
#模型训练
model.fit(X_train,
         y_train,
          batch_size=batch_size,
          validation_data=(X_test, y_test),
          epochs=epochs,callbacks=[tensorboard, checkpoint])

# 测试
test_data_filename = "/nls-kdd/datasest/KDDTest+.csv"#换成你自己的文件路径
X_test1,y_test1=load_data(test_data_filename)
smo = SMOTE(random_state=42)
X_test1, y_test1 = smo.fit_resample(X_test1, y_test1)
scaler = MinMaxScaler().fit(X_test1)
X_test1 = scaler.transform(X_test1)
X_test1=np.array(X_test1)
X_test1=np.reshape(X_test1,(X_test1.shape[0],X_test1.shape[1],1))
y_pred = model.predict_classes(X_test1)
y_test1 = np.array(y_test1)
print(y_pred)
print(y_pred.shape)
loss, accuracy = model.evaluate(X_test1, y_test1)
print("\nLoss: %.2f, Accuracy: %.2f%%" % (loss, accuracy*100))
accuracy_train = accuracy_score(y_train, model.predict_classes(X_train))
accuracy_test = accuracy_score(y_test1, model.predict_classes(X_test1))
print('\nTrain Accuracy:{: .2f}%'.format(accuracy_train*100))
print('Test Accuracy:{: .2f}%'.format(accuracy_test*100))
accuracy_test = accuracy_score(y_test1, y_pred)
f1_test=f1_score(y_test1, y_pred)
prs_test=precision_score(y_test1, y_pred)
recall = recall_score(y_test1, y_pred )
print('\nTrain Accuracy:{: .2f}%'.format(accuracy_train*100))
print('Test Accuracy:{: .2f}%'.format(accuracy_test*100))
print('Test f1_score:{: .2f}%'.format(f1_test*100))
print('Test precision:{: .2f}%'.format(prs_test*100))
print('Test recall:{: .2f}%'.format(recall*100))