分布式tensorflow学习踩坑

环境：
macOS 10.15.4
python 3.7.4
spark 3.0.2_preview
tensorflow 2.1.0
temsorflowONspark

andrew@1macbook:~/tensorflowonspark/TensorFlowOnSpark/examples/resnet#${SPARK_HOME}/bin/spark-submit --master ${MASTER} --conf spark.cores.max=${TOTAL_CORES} --conf spark.task.cpus=${CORES_PER_WORKER} --py-files ${TFoS_HOME}/examples/resnet/resnet_cifar_dist.py ${TFoS_HOME}/examples/resnet/resnet_cifar_spark.py --cluster_size ${SPARK_WORKER_INSTANCES} --epochs 1 --data_dir ${CIFAR_DATA} --num_gpus=0 --ds=multi_worker_mirrored --train_epochs 1
2020-05-25 14:22:16,663 WARN util.Utils: Your hostname, macbook resolves to a loopback address: 127.0.0.1; using 192.168.21.13 instead (on interface en0)
2020-05-25 14:22:16,664 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
2020-05-25 14:22:16,965 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
=========
 IPython
=========

Tools for Interactive Computing in Python
=========================================

    A Python shell with automatic history (input and output), dynamic object
    introspection, easier configuration, command completion, access to the
    system shell and more.  IPython can also be embedded in running programs.

Usage

    ipython [subcommand] [options] [-c cmd | -m mod | file] [--] [arg] ...

    If invoked with no options, it executes the file and exits, passing the
    remaining arguments to the script, just as if you had specified the same
    command with python. You may need to specify `--` before args to be passed
    to the script, to prevent IPython from attempting to parse them. If you
    specify the option `-i` before the filename, it will enter an interactive
    IPython session after running the script, rather than exiting. Files ending
    in .py will be treated as normal Python, but files ending in .ipy can
    contain special IPython syntax (magic commands, shell expansions, etc.).

    Almost all configuration in IPython is available via the command-line. Do
    `ipython --help-all` to see all available options.  For persistent
    configuration, look into your `ipython_config.py` configuration file for
    details.

    This file is typically installed in the `IPYTHONDIR` directory, and there
    is a separate configuration directory for each profile. The default profile
    directory will be located in $IPYTHONDIR/profile_default. IPYTHONDIR
    defaults to to `$HOME/.ipython`.  For Windows users, $HOME resolves to
    C:\Users\YourUserName in most instances.

    To initialize a profile with the default configuration file, do::

      $> ipython profile create

    and start editing `IPYTHONDIR/profile_default/ipython_config.py`

    In IPython's documentation, we will refer to this directory as
    `IPYTHONDIR`, you can change its default location by creating an
    environment variable with this name and setting it to the desired path.

    For more information, see the manual available in HTML and PDF in your
    installation, or online at https://ipython.org/documentation.html.

Subcommands
-----------

Subcommands are launched as `ipython cmd [args]`. For information on using
subcommand 'cmd', do: `ipython cmd -h`.

profile
    Create and manage IPython profiles.
kernel
    Start a kernel without an attached frontend.
locate
    print the path to the IPython dir
history
    Manage the IPython history database.
qtconsole
    DEPRECATED, Will be removed in IPython 6.0 : Launch the Jupyter Qt Console.
notebook
    DEPRECATED, Will be removed in IPython 6.0 : Launch the Jupyter HTML Notebook Server.
console
    DEPRECATED, Will be removed in IPython 6.0 : Launch the Jupyter terminal-based Console.
nbconvert
    DEPRECATED, Will be removed in IPython 6.0 : Convert notebooks to/from other formats.
trust
    DEPRECATED, Will be removed in IPython 6.0 : Sign notebooks to trust their potentially unsafe contents at load.
kernelspec
    DEPRECATED, Will be removed in IPython 6.0 : Manage Jupyter kernel specifications.
install-nbextension
    DEPRECATED, Will be removed in IPython 6.0 : Install Jupyter notebook extension files

Options
-------

Arguments that take values are actually convenience aliases to full
Configurables, whose aliases are listed on the help line. For more information
on full configurables, see '--help-all'.

--debug
    set log level to logging.DEBUG (maximize logging output)
--quiet
    set log level to logging.CRITICAL (minimize logging output)
--init
    Initialize profile with default config files.  This is equivalent
    to running `ipython profile create <profile>` prior to startup.
--autoindent
    Turn on autoindenting.
--no-autoindent
    Turn off autoindenting.
--automagic
    Turn on the auto calling of magic commands. Type %%magic at the
    IPython  prompt  for  more information.
--no-automagic
    Turn off the auto calling of magic commands.
--pdb
    Enable auto calling the pdb debugger after every exception.
--no-pdb
    Disable auto calling the pdb debugger after every exception.
--pprint
    Enable auto pretty printing of results.
--no-pprint
    Disable auto pretty printing of results.
--color-info
    IPython can display information about objects via a set of functions,
    and optionally can use colors for this, syntax highlighting
    source code and various other elements. This is on by default, but can cause
    problems with some pagers. If you see such problems, you can disable the
    colours.
--no-color-info
    Disable using colors for info related things.
--nosep
    Eliminate all spacing between prompts.
--pylab
    Pre-load matplotlib and numpy for interactive use with
    the default matplotlib backend.
--matplotlib
    Configure matplotlib for interactive use with
    the default matplotlib backend.
--autoedit-syntax
    Turn on auto editing of files with syntax errors.
--no-autoedit-syntax
    Turn off auto editing of files with syntax errors.
--simple-prompt
    Force simple minimal prompt using `raw_input`
--no-simple-prompt
    Use a rich interactive prompt with prompt_toolkit
--banner
    Display a banner upon starting IPython.
--no-banner
    Don't display a banner upon starting IPython.
--confirm-exit
    Set to confirm when you try to exit IPython with an EOF (Control-D
    in Unix, Control-Z/Enter in Windows). By typing 'exit' or 'quit',
    you can force a direct exit without any confirmation.
--no-confirm-exit
    Don't prompt the user when exiting.
--term-title
    Enable auto setting the terminal title.
--no-term-title
    Disable auto setting the terminal title.
--classic
    Gives IPython a similar feel to the classic Python prompt.
--quick
    Enable quick startup with no config files.
-i
    If running code from the command line, become interactive afterwards.
    It is often useful to follow this with `--` to treat remaining flags as
    script arguments.
--profile-dir=<Unicode> (ProfileDir.location)
    Default: ''
    Set the profile location directly. This overrides the logic used by the
    `profile` option.
--profile=<Unicode> (BaseIPythonApplication.profile)
    Default: 'default'
    The IPython profile to use.
--ipython-dir=<Unicode> (BaseIPythonApplication.ipython_dir)
    Default: ''
    The name of the IPython directory. This directory is used for logging
    configuration (through profiles), history storage, etc. The default is
    usually $HOME/.ipython. This option can also be specified through the
    environment variable IPYTHONDIR.
--log-level=<Enum> (Application.log_level)
    Default: 30
    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
    Set the log level by value or name.
--config=<Unicode> (BaseIPythonApplication.extra_config_file)
    Default: ''
    Path to an extra config file to load.
    If specified, load this config file in addition to any other IPython config.
--autocall=<Enum> (InteractiveShell.autocall)
    Default: 0
    Choices: (0, 1, 2)
    Make IPython automatically call any callable object even if you didn't type
    explicit parentheses. For example, 'str 43' becomes 'str(43)' automatically.
    The value can be '0' to disable the feature, '1' for 'smart' autocall, where
    it is not applied if there are no more arguments on the line, and '2' for
    'full' autocall, where all callable objects are automatically called (even
    if no arguments are present).
--colors=<CaselessStrEnum> (InteractiveShell.colors)
    Default: 'Neutral'
    Choices: ['Neutral', 'NoColor', 'LightBG', 'Linux']
    Set the color scheme (NoColor, Neutral, Linux, or LightBG).
--logfile=<Unicode> (InteractiveShell.logfile)
    Default: ''
    The name of the logfile to use.
--logappend=<Unicode> (InteractiveShell.logappend)
    Default: ''
    Start logging to the given file in append mode. Use `logfile` to specify a
    log file to **overwrite** logs to.
-c <Unicode> (InteractiveShellApp.code_to_run)
    Default: ''
    Execute the given command string.
-m <Unicode> (InteractiveShellApp.module_to_run)
    Default: ''
    Run the module as a script.
--ext=<Unicode> (InteractiveShellApp.extra_extension)
    Default: ''
    dotted module name of an IPython extension to load.
--gui=<CaselessStrEnum> (InteractiveShellApp.gui)
    Default: None
    Choices: ['glut', 'gtk', 'gtk2', 'gtk3', 'osx', 'pyglet', 'qt', 'qt4', 'qt5', 'tk', 'wx', 'gtk2', 'qt4']
    Enable GUI event loop integration with any of ('glut', 'gtk', 'gtk2',
    'gtk3', 'osx', 'pyglet', 'qt', 'qt4', 'qt5', 'tk', 'wx', 'gtk2', 'qt4').
--pylab=<CaselessStrEnum> (InteractiveShellApp.pylab)
    Default: None
    Choices: ['auto', 'agg', 'gtk', 'gtk3', 'inline', 'ipympl', 'nbagg', 'notebook', 'osx', 'pdf', 'ps', 'qt', 'qt4', 'qt5', 'svg', 'tk', 'widget', 'wx']
    Pre-load matplotlib and numpy for interactive use, selecting a particular
    matplotlib backend and loop integration.
--matplotlib=<CaselessStrEnum> (InteractiveShellApp.matplotlib)
    Default: None
    Choices: ['auto', 'agg', 'gtk', 'gtk3', 'inline', 'ipympl', 'nbagg', 'notebook', 'osx', 'pdf', 'ps', 'qt', 'qt4', 'qt5', 'svg', 'tk', 'widget', 'wx']
    Configure matplotlib for interactive use with the default matplotlib
    backend.
--cache-size=<Int> (InteractiveShell.cache_size)
    Default: 1000
    Set the size of the output cache.  The default is 1000, you can change it
    permanently in your config file.  Setting it to 0 completely disables the
    caching system, and the minimum value accepted is 3 (if you provide a value
    less than 3, it is reset to 0 and a warning is issued).  This limit is
    defined because otherwise you'll spend more time re-flushing a too small
    cache than working

To see all available configurables, use `--help-all`

Examples
--------

    ipython --matplotlib       # enable matplotlib integration
    ipython --matplotlib=qt    # enable matplotlib integration with qt4 backend
    
    ipython --log-level=DEBUG  # set logging to DEBUG
    ipython --profile=foo      # start with profile foo
    
    ipython profile create foo # create profile foo w/ default config files
    ipython help profile       # show the help for the profile subcmd
    
    ipython locate             # print the path to the IPython directory
    ipython locate profile foo # print the path to the directory for profile `foo`

[TerminalIPythonApp] CRITICAL | Bad config encountered during initialization:
[TerminalIPythonApp] CRITICAL | Unrecognized flag: '--cluster_size'
2020-05-25 14:22:17,571 INFO util.ShutdownHookManager: Shutdown hook called
2020-05-25 14:22:17,572 INFO util.ShutdownHookManager: Deleting directory /private/var/folders/y6/_yfkw8qs5t31xb_4x0d6g3s40000gp/T/spark-dadf8aec-f955-424b-8e61-dd0c312bc621
2020-05-25 14:22:17,577 INFO util.ShutdownHookManager: Deleting directory /private/var/folders/y6/_yfkw8qs5t31xb_4x0d6g3s40000gp/T/localPyFiles-34e8001c-fd7e-4c66-ba24-e54754c08cd9

报错原因：

一个参数错误：
CRITICAL | Unrecognized flag: ‘–cluster_size’
集群的大小错误