Deeplearning4j_deeplearning4j stringtocategoricaltransform-CSDN博客

本文链接：https://blog.csdn.net/lifeqiuzhi521/article/details/80904009

Tutorials Introduction to Deep Learning Neural Networks Data & ETL Tuning & Training Advanced Usage Open-Source Community Natural Language Processing Keras Model Import ND4J: Numpy for the JVM More Resources Other Languages

What Is DeepLearning4j?

Deeplearning4j is a Java-based toolkit for building, training and deploying deep neural networks, the regressions and KNN.

Deeplearning4j Components

Deeplearning4j has the following sub-projects.

DataVec performs data ingestion, normalization and transformation into feature vectors
Deeplearning4j provides tools to configure neural networks and build computation graphs
DL4J-Examples contains working examples for classification and clustering of images, time series and text.
Keras Model Import helps import trained models from Python and Keras to DeepLearning4J and Java.
ND4J allows Java to access Native Libraries to quickly process Matrix Data on CPUs or GPUs.
ScalNet is a Scala wrapper for Deeplearning4j inspired by Keras. Runs on multi-GPUs with Spark.
RL4J implements Deep Q Learning, A3C and other reinforcement learning algorithms for the JVM.
Arbiter helps search the hyperparameter space to find the best neural net configuration.

Ingesting, cleaning, joining, scaling, normalizing and transforming data are jobs that must be done in any sort of data analysis. This work may not be exciting, but it’s a precondition of deep learning. DataVec is our toolkit for that process. We give data scientists and developers tools to turn raw data such as images, video, audio, text and time series into feature vectors for neural nets.

Github Repo

The DataVec Github repo is here. Here is how the repo is organized.

datavec-api : rules for preprocessing data and defining data pipelines.
datavec-data : knows how to understand audio, video, images, text data types
datavec-spark : runs distributed data pipelines on Spark
datavec-local : runs Datavec standalone on desktop. For inference.
datavec-camel : connects to external Camel components. Camel allows you to define routes and integrates with many data sources. DataVec-camel sends data to datavec as a destination from whichever Camel source you specify.

DataVec Examples

There are DataVec examples in our examples repo on Github.

A descriptive summary of many of the examples is here.

JavaDoc

Here is the DataVec JavaDoc.

DataVec overview

Neural Networks process multi-dimensional arrays of numerical data. Getting your data from a CSV file, or a directory of images, to be serialized into numeric arrays is the job of DataVec. DataVec is an ETL tool specifically built for machine learning pipelines.

DataVec: Commonly used classes

Here’s a list of some important DataVec classes:

Input Split

Splitting data into Test and Train

InputSplit.sample to split data into Test and Train

Randomize Data

FileSplit.random to randomize data

Base class for reading and serializing data. RecordReaders ingest your data input and return a List of Serializable objects (Writables).

RecordReader

Implementations of particular RecordReaders:

CSVRecordReader for CSV data
CSVNLinesSequenceRecordReader for Sequence Data
ImageRecordReader for images
JacksonRecordReader for JSON data
RegexLineRecordReader for parsing log files
WavFileRecordReader for audio files
LibSvmRecordReader for Support Vector Machine
VideoRecordReader for reading Video

For re-organizing, joining, normalizing and transforming data.

Transform

Specific transform implementations

CategoricalToIntegerTransform to convert category names to integers
CategoricalToOneHotTransform convert catagory name to onehot representation
ReorderColumnsTransform rearrange columns
RenameColumnsTransform rename columns
StringToTimeTransform convert timestring

The labels for data input may be based on the directory where the image is stored.

ParentPathLabelGenerator Label based on parent directory
PatternPathLabelGenerator Derives label based on a string within the file path

DataNormalization

Normalizer

DeepLearning4J

Deeplearning4j is where you design your neural networks. It is a domain specific language (DSL) for configuring neural networks.

Github Repo

The Deeplearning4j Github repo is here. Here’s how the repo is organized.

deeplearning4j-core : datasetiterators and everything you need to run dl4j on the desktop.
deeplearning4j-cuda : cudnn and anything cuda specific.
deeplearning4j-graph : graph processing for deepwalk.
deeplearning4j-modelimport: this imports neural net models from Keras, which in turn can import models from major frameworks like Theano, Tensorflow, Caffe and CNTK
deeplearning4j-nlp-parent: text analytics for English, Japanese and Korean as well as external tokenizers and plugins to toolsets like like UIMA, which itself performs dependency parsing, semantic role labeling, relation extraction and QA systems. We integrate with toolsets like UIKMA to pass stuff to word2vec.
nlp: Word2vec, doc2vec and other tools.
deeplearning4j-nn : a pared-down neural net DSL with fewer dependencies. Configures multilayer nets with a builder pattern for setting hyperparameters.
deeplearning4j-scaleout : AWS provisioning, parallelwrapper desktop parameter averaging (single box 96 cores) so you don’t have to run spark if you don’t want to; one for parameter server and the other not; streaming is kafka and spark streaming; spark is spark training and nlp on spark: dist. word2vec
deeplearning4j-ui-parent : neural net training heuristics and visualization

JavaDoc

Here is the Deeplearning4j JavaDoc here.

DeepLearning4J Examples

There are Deeplearning4j examples in the Github repository here.

A descriptive summary of many of the examples is here.

Deeplearning4j: Frequently used classes

MultiLayerConfiguration Configure a network
MultiLayerConfiguration.Builder Builder interface to configure a network
MultiLayerNetwork Builds a Network from the configuration
ComputationGraph Builds a Computation Graph style Network
ComputationGraphConfiguration Configuration for Computation Graph
ComputationGraphConfiguration.GraphBuilder Builder interface for Computation Graph configuration
UiServer Adds a web based Gui to view training parameter progress and configuration of Network

Keras Model Import

If you have worked with the Python Library Keras and would like to import a trained model, or a model configuration, into Deeplearning4j, please see our model import feature.

Github Repo

The Model Import is actually part of DeepLearning4J, but it is worth its own section. Github folder is here.

Model Import Examples

We will add examples here

Model Import Frequently Used Classes

KerasModel.Import saved Keras Model to DeepLearning4J MultiLayerNetwork or Computation Graph

Video

Here’s a video showing how to import of a Keras model to DL4J:

ND4J

ND4J is the numerical computing library that underpins Deeplearning4j. It is a tensor library, the JVM’s answer to Numpy.

Github Repo

Here is the ND4J Github repo. ND4J is a DSL for handling n-dimensional arrays (NDArrays), also known as tensors.

nd4j-parameter-server-parent : a robust parameter server for distributed neural net training using Aeron.
nd4j-backends : hardware specific backends the optimize on GPUs and CPUs.

JavaDoc

Here is the ND4J JavaDoc.

ND4J Examples

Here are ND4J examples.

ND4J: Frequently Used Classes

You may not use some of these classes directly, but when you configure a neural network. Behind the scenes, the configurations you set for OptimizationAlgorithm, Updater, and LossFunction are all done in ND4J.

DataSetPreProcessor tools for Normalizing an image or numerical data
BaseTransformOp Activation functions, tanh, sigmoid, relu, Softmax …
GradientUpdater Stochastic Gradient Descent, AdaGrad, Adam, Nesterovs ..

ScalNet

ScalNet is Keras for Scala. It’s a Scala wrapper for Deeplearning4j that can run Spark on multi-GPUs.

Github Repo

ScalNet on Github

RL4J

RL4J is a library and environment for reinforcement learning on the JVM. It includes Deep Q learning, A3C and other algorithms implemented in Java and integrated with DL4J and ND4J.

Github Repo

Arbiter

Arbiter helps you search the hyperparameter space to find the best tuning and architecture for a neural net. This is important because finding the right architecture and hyperparamters is a very large combinatorial problem. The winning ImageNet teams at corporate labs like Microsoft are searching through hyperparameters to surface 150-layer networks like ResNet. Arbiter includes grid search, random search, some Bayesian methods, as well as model evaluation tools.

Github Repo

Here is the Arbiter Github repository.

arbiter-core : Arbiter-core searches the hyperparameter space with algorithms like grid search. Provides a GUI.
arbiter-deeplearning4j : Arbiter can talk to DL4J models. When you do model search, you need to be able to run the model. This pilots the model and finds the best model.