InternalError: Failed to create session.錯誤及解決方式

前言

這個錯誤是在一開始訓練Keras（使用TensorFlow當backend）模型的時候就出現。
Python版本：3.5.2
Keras版本：2.1.3
TensorFlow版本：1.9.0

錯誤訊息

InternalErrorTraceback (most recent call last)
<ipython-input-4-d4cc2ca313a3> in <module>
10 model.compile(loss=‘mse’, optimizer=‘adam’)
11 # fit network
—> 12 history = model.fit(X_train, y_train, epochs=3000, batch_size=16, validation_data=(x_test, y_test), verbose=2, shuffle=False)
13 #history = model.fit(X,y, epochs=3000, batch_size=16, verbose=2, shuffle=False)
14 # plot history
/usr/local/lib/python3.5/dist-packages/keras/models.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
963 initial_epoch=initial_epoch,
964 steps_per_epoch=steps_per_epoch,
–> 965 validation_steps=validation_steps)
966
967 def evaluate(self, x=None, y=None,
/usr/local/lib/python3.5/dist-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
1667 initial_epoch=initial_epoch,
1668 steps_per_epoch=steps_per_epoch,
-> 1669 validation_steps=validation_steps)
1670
1671 def evaluate(self, x=None, y=None,
/usr/local/lib/python3.5/dist-packages/keras/engine/training.py in _fit_loop(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)
1204 ins_batch[i] = ins_batch[i].toarray()
1205
-> 1206 outs = f(ins_batch)
1207 if not isinstance(outs, list):
1208 outs = [outs]
/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py in call(self, inputs)
2471 feed_dict[tensor] = value
2472 fetches = self.outputs + [self.updates_op] + self.fetches
-> 2473 session = get_session()
2474 updated = session.run(fetches=fetches, feed_dict=feed_dict,
2475 **self.session_kwargs)
/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py in get_session()
174 config = tf.ConfigProto(intra_op_parallelism_threads=num_thread,
175 allow_soft_placement=True)
–> 176 _SESSION = tf.Session(config=config)
177 session = _SESSION
178 if not _MANUAL_VAR_INIT:
/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py in init(self, target, graph, config)
1561
1562 “”"
-> 1563 super(Session, self).init(target, graph, config=config)
1564 # NOTE(mrry): Create these on first __enter__ to avoid a reference cycle.
1565 self._default_graph_context_manager = None
/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py in init(self, target, graph, config)
631 if self._created_with_new_api:
632 # pylint: disable=protected-access
–> 633 self._session = tf_session.TF_NewSession(self._graph._c_graph, opts)
634 # pylint: enable=protected-access
635 else:
InternalError: Failed to create session.

解決方法

到命令行使用nvidia-smi查看GPU memory的使用情況：

發現GPU memory己被其它的程序佔滿。
試著將這些程序關掉，然後再重新運行剛剛那段代碼（不必重啟kernel），發現問題成功解決！