Python code standard

• Module Level Dunder Names

   Module level "dunders" (i.e. names with two leading and two trailing underscores) such as __all__, __author__, __version__, etc.
   should be placed after the module docstring but before any import statements except from __future__ imports.
   Python mandates that future-imports must appear in the module before any other code except docstrings:
  

• Whitespace in Expressions and Statements Pet Peeves

Avoid extraneous whitespace in the following situations:

• Immediately inside parentheses, brackets or braces.

   Yes: spam(ham[1], {eggs: 2})
   No:  spam( ham[ 1 ], { eggs: 2 } )
  

• Between a trailing comma and a following close parenthesis.

   Yes: foo = (0,)
   No:  bar = (0, )
  

• Immediately before a comma, semicolon, or colon:

   Yes: if x == 4: print x, y; x, y = y, x
   No:  if x == 4 : print x , y ; x , y = y , x
  

• However, in a slice the colon acts like a binary operator, and should have equal amounts on either side (treating it as the operator with the lowest priority). In an extended slice, both colons must have the same amount of spacing applied. Exception: when a slice parameter is omitted, the space is omitted.

   Yes:
   
   ham[1:9], ham[1:9:3], ham[:9:3], ham[1::3], ham[1:9:]
   ham[lower:upper], ham[lower:upper:], ham[lower::step]
   ham[lower+offset : upper+offset]
   ham[: upper_fn(x) : step_fn(x)], ham[:: step_fn(x)]
   ham[lower + offset : upper + offset]
   No:
   
   ham[lower + offset:upper + offset]
   ham[1: 9], ham[1 :9], ham[1:9 :3]
   ham[lower : : upper]
   ham[ : upper]
  

• Immediately before the open parenthesis that starts the argument list of a function call:

   Yes: spam(1)
   No:  spam (1)
  

• Immediately before the open parenthesis that starts an indexing or slicing:

   Yes: dct['key'] = lst[index]
   No:  dct ['key'] = lst [index]
  

• More than one space around an assignment (or other) operator to align it with another.

   Yes:
   
   x = 1
   y = 2
   long_variable = 3
   No:
   
   x             = 1
   y             = 2
   long_variable = 3
  

• Indentation

Use 4 spaces per indentation level.

Continuation lines should align wrapped elements either vertically using Python's implicit line joining inside parentheses, brackets and braces, or using a hanging indent [7]. When using a hanging indent the following should be considered; there should be no arguments on the first line and further indentation should be used to clearly distinguish itself as a continuation line.

Yes:

# Aligned with opening delimiter.
foo = long_function_name(var_one, var_two,
                         var_three, var_four)

# More indentation included to distinguish this from the rest.
def long_function_name(
        var_one, var_two, var_three,
        var_four):
    print(var_one)

# Hanging indents should add a level.
foo = long_function_name(
    var_one, var_two,
    var_three, var_four)
No:

# Arguments on first line forbidden when not using vertical alignment.
foo = long_function_name(var_one, var_two,
    var_three, var_four)

# Further indentation required as indentation is not distinguishable.
def long_function_name(
    var_one, var_two, var_three,
    var_four):
    print(var_one)
The 4-space rule is optional for continuation lines.

Optional:

# Hanging indents *may* be indented to other than 4 spaces.
foo = long_function_name(
  var_one, var_two,
  var_three, var_four)
When the conditional part of an if-statement is long enough to require that it be written across multiple lines, it's worth noting that the combination of a two character keyword (i.e. if), plus a single space, plus an opening parenthesis creates a natural 4-space indent for the subsequent lines of the multiline conditional. This can produce a visual conflict with the indented suite of code nested inside the if-statement, which would also naturally be indented to 4 spaces. This PEP takes no explicit position on how (or whether) to further visually distinguish such conditional lines from the nested suite inside the if-statement. Acceptable options in this situation include, but are not limited to:

# No extra indentation.
if (this_is_one_thing and
    that_is_another_thing):
    do_something()

# Add a comment, which will provide some distinction in editors
# supporting syntax highlighting.
if (this_is_one_thing and
    that_is_another_thing):
    # Since both conditions are true, we can frobnicate.
    do_something()

# Add some extra indentation on the conditional continuation line.
if (this_is_one_thing
        and that_is_another_thing):
    do_something()
(Also see the discussion of whether to break before or after binary operators below.)

The closing brace/bracket/parenthesis on multiline constructs may either line up under the first non-whitespace character of the last line of list, as in:

my_list = [
    1, 2, 3,
    4, 5, 6,
    ]
result = some_function_that_takes_arguments(
    'a', 'b', 'c',
    'd', 'e', 'f',
    )
or it may be lined up under the first character of the line that starts the multiline construct, as in:

my_list = [
    1, 2, 3,
    4, 5, 6,
]
result = some_function_that_takes_arguments(
    'a', 'b', 'c',
    'd', 'e', 'f',
)

• Tabs or Spaces?

   Spaces are the preferred indentation method.
   Tabs should be used solely to remain consistent with code that is already indented with tabs.
   Python 3 disallows mixing the use of tabs and spaces for indentation.
   Python 2 code indented with a mixture of tabs and spaces should be converted to using spaces exclusively.
   When invoking the Python 2 command line interpreter with the -t option, it issues warnings about code that illegally mixes tabs and spaces. When using -tt these warnings become errors. These options are highly recommended!
  

• Maximum Line Length

    Limit all lines to a maximum of 79 characters.
    For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.
    The Python standard library is conservative and requires limiting lines to 79 characters (and docstrings/comments to 72).
    The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses. These should be used in preference to using a backslash for line continuation.
    Backslashes may still be appropriate at times. For example, long, [multiple with-statements](https://www.python.org/dev/peps/pep-0008/#multiline-if-statements) cannot use implicit continuation, so backslashes are acceptable:
    
    with open('/path/to/some/file/you/want/to/read') as file_1, \
        open('/path/to/some/file/being/written', 'w') as file_2:
       file_2.write(file_1.read())
       
   (See the previous discussion on multiline if-statements for further thoughts on the indentation of such multiline with-statements.)
   Another such case is with assert statements.
   Make sure to indent the continued line appropriately.
  

• Should a Line Break Before or After a Binary Operator?

Following the tradition from mathematics usually results in more readable code:

# Yes: easy to match operators with operands
income = (gross_wages
          + taxable_interest
          + (dividends - qualified_dividends)
          - ira_deduction
          - student_loan_interest)
In Python code, it is permissible to break before or after a binary operator, as long as the convention is consistent locally. For new code Knuth's style is suggested.

• Blank Lines

  Surround top-level function and class definitions with two blank lines.
  
  Method definitions inside a class are surrounded by a single blank line.
  
  Extra blank lines may be used (sparingly) to separate groups of related functions. Blank lines may be omitted between a bunch of related one-liners (e.g. a set of dummy implementations).
  
  Use blank lines in functions, sparingly, to indicate logical sections.
  
  Python accepts the control-L (i.e. ^L) form feed character as whitespace; Many tools treat these characters as page separators, so you may use them to separate pages of related sections of your file. Note, some editors and web-based code viewers may not recognize control-L as a form feed and will show another glyph in its place.
  

• Use single quotes

   Use single-quotes for string literals, e.g. 'my-identifier', but use double-quotes for strings that are likely to contain single-quote characters as part of the string itself (such as error messages, or any strings containing natural language), e.g. "You've got an error!".
  
  Single-quotes are easier to read and to type, but if a string contains single-quote characters then double-quotes are better than escaping the single-quote characters or wrapping the string in double single-quotes.
  
  We also use triple single-quotes for docstrings, see http://docs.ckan.org/en/latest/contributing/python.html#docstrings.
  

• Imports

  • Avoid creating circular imports by only importing modules more specialized than the one you are editing.

    CKAN often uses code imported into a data structure instead of importing names directly. For example CKAN controllers only use get_action to access logic functions. This allows customization by CKAN plugins.
    

  • Don’t use from module import *. Instead list the names you need explicitly:

      from module import name1, name2
    

  Use parenthesis around the names if they are longer than one line:

   	from module import (name1, name2, ...
   name12, name13)
  

  • Most of the current CKAN code base imports just the modules and then accesses names with module.name(http://module.name/). This allows circular imports in some cases and may still be necessary for exsiting code, but is not recommended for new code.

  • Make all imports at the start of the file, after the module docstring. Imports should be grouped in the following order:

  Standard library imports
  Third-party imports
  CKAN imports

• Logging

We use the Python standard library’s logging module to log messages in CKAN, e.g.:
import logging
...
logger = logging.getLogger(__name__)
...
logger.debug('some debug message')
When logging:

Keep log messages short.
Don’t include object representations in the log message. It is useful to include a domain model identifier where appropriate.

Choose an appropriate log-level (DEBUG, INFO, ERROR, WARNING or CRITICAL, see Python’s Logging HOWTO).

• String formatting
  Don’t use the old %s style string formatting, e.g. “i am a %s” % sub. This kind of string formatting is not helpful for internationalization.

  Use the new .format() method instead, and give meaningful names to each replacement field, for example:
  _(’ … {foo} … {bar} …’).format(foo=‘foo-value’, bar=‘bar-value’)

  CKAN strives to only use Unicode internally (via the unicode type) and to convert to/from ASCII at the interface to other systems and libraries if necessary.

• Docstrings
  We want CKAN’s docstrings to be clear and easy to read for programmers who are smart and competent but who may not know a lot of CKAN technical jargon and whose first language may not be English. We also want it to be easy to maintain the docstrings and keep them up to date with the actual behaviour of the code as it changes over time. So:

  All modules and all public functions, classes and methods exported by a module should normally have docstrings (see PEP 257).

  - Keep docstrings short, describe only what’s necessary and no more.
  - Keep docstrings simple: use plain, concise English.
  - Try to avoid repetition.
  

• PEP 257 (Docstring Conventions)
  Generally, follow PEP 257 for docstrings. We’ll only describe the ways that CKAN differs from or extends PEP 257 below.

  CKAN docstrings deviate from PEP 257 in a couple of ways:

  • We use ‘’‘triple single quotes’’’ around docstrings, not “”“triple double quotes”"" (put triple single quotes around one-line docstrings as well as multi-line ones, it makes them easier to expand later)
  • We use Sphinx domain object cross-references to cross-reference to other code objects (see below)
  • We use Sphinx directives for documenting parameters, exceptions and return values (see below)

• Referencing other code objects with :py:

If you want to refer to another Python or JavaScript module, function or class etc. in a docstring (or from a .rst file), use Sphinx domain object cross-references, for example:

See :py:mod:`ckan.lib.helpers`.

See :py:func:`ckan.logic.action.create.package_create`.

See :py:class:`ckan.logic.NotFound`.

For the full list of types of cross-reference, see the Sphinx docs.

Note

These kinds of cross-references can also be used to reference other types of object besides Python objects,

for example JavaScript objects or even command-line scripts and options and environment variables. See the Sphinx docs for the full details.

Cross-referencing objects like this means that Sphinx will style the reference with the right CSS, and hyperlink the reference to the docs for the referenced object.

Sphinx can also generate error messages when non-existent objects are referenced, which helps to keep the docs up to date as the code changes.

Tip

Sphinx will render a cross-reference like :py:func:`ckan.logic.action.create.package_create` as the full name of the function: ckan.logic.action.create.package_create().

If you want the docs to contain only the local name of the function (e.g. just package_create()), put a ~ at the start:

:py:func:`~ckan.logic.action.create.package_create` 
(But you should always use the fully qualified name in your docstring or *.rst file.)

• Documenting exceptions raised with :raises

There are a few guidelines that CKAN code should follow regarding exceptions:

All public functions that CKAN exports for third-party code to use should document any exceptions they raise. See below for how to document exceptions raised.

For example the template helper functions in ckan.lib.helpers, anything imported into ckan.plugins.toolkit, and all of the action API functions defined in ckan.logic.action, should list exceptions raised in their docstrings.

This is because CKAN themes, extensions and API clients need to be able to call CKAN code without crashing, so they need to know what exceptions they should handle (and extension developers shouldn’t have to understand the CKAN core source code).

On the other hand, internal functions that are only used within CKAN shouldn’t list exceptions in their docstrings.

This is because it would be difficult to keep all the exception lists up to date with the actual code behaviour, so the docstrings would become more misleading than useful.

Code should only raise exceptions from within its allowed set.

Each module in CKAN has a set of zero or more exceptions, defined somewhere near the module, that code in that module is allowed to raise. For example ckan/logic/__init__.py defines a number of exception types for code in ckan/logic/ to use. CKAN code should never raise exceptions types defined elsewhere in CKAN, in third-party code or in the Python standard library.

All code should catch any exceptions raised by called functions, and either handle the exception, re-raise the exception (if it’s from the code’s set of allowed exception types), or wrap the exception in an allowed exception type and re-raise it.

This is to make it easy for a CKAN core developer to look at the source code of an internal function, scan it for the keyword raise, and see what types of exception the function may raise, so they know what exceptions they need to catch if they’re going to call the function. Developers shouldn’t have to read the source of all the functions that a function calls (and the functions they call…) to find out what exceptions they needs to catch to call a function without crashing.



Use :raises: to document exceptions raised by public functions. The docstring should say what type of exception is raised and under what conditions. Use :py:class: to reference exception types. For example:

def member_list(context, data_dict=None):
 '''Return the members of a group.

 ... (parameters and return values documented here) ...

 :raises: :py:class:`ckan.logic.NotFound`: if the group doesn't exist

 '''

• Sphinx field lists

Use Sphinx field lists for documenting the parameters, exceptions and returns of functions:

Use :param and :type to describe each parameter
Use :returns and :rtype to describe each return
Use :raises to describe each exception raised
Example of a short docstring:

@property
def packages(self):
    '''Return a list of all packages that have this tag, sorted by name.

 :rtype: list of ckan.model.package.Package objects

 '''
Example of a longer docstring:

@classmethod
def search_by_name(cls, search_term, vocab_id_or_name=None):
    '''Return all tags whose names contain a given string.

 By default only free tags (tags which do not belong to any vocabulary)
 are returned. If the optional argument ``vocab_id_or_name`` is given
 then only tags from that vocabulary are returned.

 :param search_term: the string to search for in the tag names
 :type search_term: string
 :param vocab_id_or_name: the id or name of the vocabulary to look in
 (optional, default: None)
 :type vocab_id_or_name: string

 :returns: a list of tags that match the search term
 :rtype: list of ckan.model.tag.Tag objects

 '''
The phrases that follow :param foo:, :type foo:, or :returns: should not start with capital letters or end with full stops. These should be short phrases and not full sentences. If more detail is required put it in the function description instead.

Indicate optional arguments by ending their descriptions with (optional) in brackets. Where relevant also indicate the default value: (optional, default: 5).

You can also use a little inline reStructuredText markup in docstrings, e.g. *stars for emphasis* or ``double-backticks for literal text``

• Action API docstrings

Docstrings from CKAN’s action API are processed with autodoc and included in the API chapter of CKAN’s documentation. The intended audience of these docstrings is users of the CKAN API and not (just) CKAN core developers.

In the Python source each API function has the same two arguments (context and data_dict), but the docstrings should document the keys that the functions read from data_dict and not contextand data_dict themselves, as this is what the user has to POST in the JSON dict when calling the API.

Where practical, it’s helpful to give examples of param and return values in API docstrings.

CKAN datasets used to be called packages and the old name still appears in the source, e.g. in function names like package_list(). When documenting functions like this write dataset not package, but the first time you do this put package after it in brackets to avoid any confusion, e.g.

def package_show(context, data_dict):
 '''Return the metadata of a dataset (package) and its resources.
Example of a ckan.logic.action API docstring:

def vocabulary_create(context, data_dict):
 '''Create a new tag vocabulary.

 You must be a sysadmin to create vocabularies.

 :param name: the name of the new vocabulary, e.g. ``'Genre'``
 :type name: string
 :param tags: the new tags to add to the new vocabulary, for the format of
 tag dictionaries see ``tag_create()``
 :type tags: list of tag dictionaries

 :returns: the newly-created vocabulary
 :rtype: dictionary

 '''

• Other Recommendations
  https://www.python.org/dev/peps/pep-0008/#other-recommendations

There are various tools that can help you to check your Python code for PEP8 conformance and general code quality. We recommend using them:

pep8 checks your Python code against some of the style conventions in PEP 8. As mentioned above, only perform style clean-ups on master to help avoid spurious merge conflicts.
pylint analyzes Python source code looking for bugs and signs of poor quality.
pyflakes also analyzes Python programs to detect errors.
flake8 combines both pep8 and pyflakes into a single tool.
Syntastic is a Vim plugin with support for flake8, pyflakes and pylint.