jupyter平台中用户如何自定义magic

Magic Command简介

Jupyter magic 命令形似%abc(line magic), %%abc(cell magic)我们常用的画图命令%matplotlib就是一种magic命令。
magic命令是对jupyter功能的扩展，使用magic能简化在jupyter做实验编写代码工作，解决重复代码操作、新人编写和理解功能模块困难等问题。

自定义Magic的方式

目前官方推荐的有“注册独立的magic方法”和“继承Magic基类”两种方式

注册独立的Magic方法

register csv cell_magic

# create a slightly more useful %%csv cell magic that parses a CSV string and returns a pandas DataFrame
 
import pandas as pd
from io import StringIO
from IPython.core.magic import (register_line_magic,
                                register_cell_magic)
 
@register_cell_magic
def csv(line, cell):
    # We create a string buffer containing the
    # contents of the cell.
    sio = StringIO(cell)
    # We use Pandas' read_csv function to parse
    # the CSV string.
    return pd.read_csv(sio)

需要注意的是：
为了避免关键词冲突，使用完该种自定义magic后需要及时清除。操作：del csv
调用实例：

继承Magic基类

通过继承magic基类的方式，你不仅能同时注册line_magic、cell_magic、line_cell_magic三种magic，而且能够让自定义magic面向所有的ipython对象，在多次调用间持久保持唤醒。
下以自定义csvmagic为例说明：
inheriting from the Magic class

%%writefile csvmagic.py
# This code can be put in any Python module, it does not require IPython
# itself to be running already.  It only creates the magics subclass but
# doesn't instantiate it yet.
from __future__ import print_function
import pandas as pd
from io import StringIO
from IPython.core.magic import (Magics, magics_class, line_magic,
                                cell_magic, line_cell_magic)
 
# The class MUST call this class decorator at creation time
@magics_class
class CsvMagics(Magics):
 
    @line_magic
    def lmagic(self, line):
        "my line magic"
        print("Full access to the main IPython object:", self.shell)
        print("Variables in the user namespace:", list(self.shell.user_ns.keys()))
        return line
 
    @cell_magic
    def csv(self, line, cell):
        # We create a string buffer containing the
        # contents of the cell.
        sio = StringIO(cell)
        # We use Pandas' read_csv function to parse
        # the CSV string.
        return pd.read_csv(sio)
 
    @line_cell_magic
    def lcmagic(self, line, cell=None):
        "Magic that works both as %lcmagic and as %%lcmagic"
        if cell is None:
            print("Called as line magic")
            return line
        else:
            print("Called as cell magic")
            return line, cell
 
 
# In order to actually use these magics, you must register them with a
# running IPython.
 
def load_ipython_extension(ipython):
    """
    Any module file that define a function named `load_ipython_extension`
    can be loaded via `%load_ext module.path` or be configured to be
    autoloaded by IPython at startup time.
    """
    # You can register the class itself without instantiating it.  IPython will
    # call the default constructor on it.
    ipython.register_magics(CsvMagics)

使用%load_ext csvmagic即可加载自定义magic模块:

使用继承Magic基类的方式，可以将自定义magic模块存放置用户根目录、~/.ipython/extensions扩展目录下、第三方工具包目录下。 不同的存放路径会导致调用magic方式上的略微不同；下一小节将会讲解各种调用方法。

调用自定Magic的方式

从用户根目录下调用

当我们在notebook中继承Magic类编写自定义magic模块时，使用%%writefile xxx.py这个magic命令默认将模块存放在用户的根目录下（目前测试集群是在/umecron/路径下）。从用户根目录下调用自定义magic模块只需要使用%load_ext xxx加载扩展magic命令即可，但是这种方法只局限于本用户使用，跨路径用户是无法直接调用的。

从ipython扩展路径下调用

不同于从用户根目录下调用，将用户自定义magic模块统一放置于 ~/.ipython/extensions/ 路径下即可，调用自定义模块依然是使用%load_ext xxx加载扩展magic命令。
需要注意的是：
Loading extensions from ~/.ipython/extensions is deprecated. We recommend managing extensions like any other Python packages, in site-packages.
官方不建议将自定义magic模块置于/extensions路径下。

ipython自启动模块调用

与ipython扩展路径类似，如果自定义magic模块放置于~/.ipython/profile_default/startup路径下，启动ipython时模块会自动调用，这是目前已知最简单、省事的操作。无需要加载模块操作在notebook直接使用magic调用语句即可：
~/.ipython/profile_default/startup/csvmagic.py

# This code can be put in any Python module, it does not require IPython
# itself to be running already.  It only creates the magics subclass but
# doesn't instantiate it yet.
from __future__ import print_function
import pandas as pd
from io import StringIO
from IPython.core.magic import (Magics, magics_class, line_magic,
                                cell_magic, line_cell_magic)
 
# The class MUST call this class decorator at creation time
@magics_class
class CsvMagics(Magics):
 
    @line_magic
    def lmagic(self, line):
        "my line magic"
        print("Full access to the main IPython object:", self.shell)
        print("Variables in the user namespace:", list(self.shell.user_ns.keys()))
        return line
 
    @cell_magic
    def csv(self, line, cell):
        # We create a string buffer containing the
        # contents of the cell.
        sio = StringIO(cell)
        # We use Pandas' read_csv function to parse
        # the CSV string.
        return pd.read_csv(sio)
 
    @line_cell_magic
    def lcmagic(self, line, cell=None):
        "Magic that works both as %lcmagic and as %%lcmagic"
        if cell is None:
            print("Called as line magic")
            return line
        else:
            print("Called as cell magic")
            return line, cell
 
ipy = get_ipython()
ipy.register_magics(CsvMagics)

从第三方工具包下调用

为了将自定义magic模块置于第三方模块路径下，你可以使用setuptools, distutils等工具。参考第三方工具包，模块路径形似：
.
├── example_magic
│ ├── init.py
│ └── csvmagic.py
└── setup.py
用户如果需要使用自定义magic，只需要同第三方工具包引用自定义模块，然后使用magic命令调用。使用这种方式能实现按需求引入自定义模块，有效避免模块冲突。使用方法如下：
制作第三方工具包demo已上传至github可以参考：https://github.com/liuweibin6566396837/spark_magic-py-master

自定义pyspark magic实践

参照自定义magic流程，定义了一个起pyspark会话的magic方便在notebook中使用pyspark。
可以使用line magic起默认配置的pyspark会话，或者用户指定基本配置参数起cell magic pyspark会话：

由于基于原有pyspark会话起新会话存在问题，目前自定义配置只支持基本配置，如果配置较复杂推荐使用SparkSession起pyspark会话，如需配置复杂会话可以参考magic源码直接起会话.

参考链接

1.https://github.com/ipython/ipython/blob/8bda98619aba4bf0aa75d92991c3f7d240d1c781/docs/source/config/custommagics.rst#defining-custom-magics