python jdict_jdict python中的javascript dict

最新推荐文章于 2023-10-30 09:00:00 发布

weixin_26711867

最新推荐文章于 2023-10-30 09:00:00 发布

阅读量173

点赞数

原文链接：https://medium.com/@anna_1580/jdict-javascript-dict-in-python-e7a5383939ab

版权

python jdict

致谢(Acknowledgment)

Thanks to Asher Sterkin, BlackSwan Technologies SVP of Engineering, who proposed the idea for this project and provided invaluable guidance on its development. The project is an offshoot of BlackSwan’s work developing a Cloud AI Operating System, or CAIOS, which is intended to provide 10x productivity improvements when coding for cloud/serverless environments.

感谢BlackSwan Technologies工程高级副总裁Asher Sterkin提出的这个项目构想，并为该项目的开发提供了宝贵的指导。该项目是黑天鹅的工作制定一个云AI操作系统或旁枝CAIOS ，其目的是编码云/无服务器环境时提供10度的生产率的提高。

问题陈述 (Problem Statement)

JavaScript has advantages over native Python when it comes to accessing attribute values in a dictionary object. In this article, we will demonstrate how to achieve the same level of usability and performance in Python as with JavaScript.

当访问字典对象中的属性值时，JavaScript优于本机Python。在本文中，我们将演示如何在Python中实现与JavaScript相同水平的可用性和性能。

JavaScript字典访问 (JavaScript Dictionary Access)

With JavaScript, key/value pairs can be accessed directly from a dictionary object either through the indexer or as a property of the object.

使用JavaScript，可以通过索引器或作为对象的属性直接从字典对象访问键/值对。

var dict = {FirstName: “Chris”, “one”: 1, 1: “some value”};// using indexervar name = dict[“FirstName”];// as propertyvar name = dict.FirstName;

In other words, in JavaScript, one could use dict.x and dict[‘x’] or dict[y] where y=’x’ interchangeably.

换句话说，在JavaScript中，可以互换使用dict.x和dict ['x']或dict [y]，其中y ='x'。

Python字典 (Python Dictionary)

Even though it is possible to access object attributes by obj.attr notation, it does not work for dictionaries.

即使可以通过obj.attr表示法访问对象属性，它也不适用于字典。

In the dictionary, you can get the value using the following methods:

在字典中，可以使用以下方法获取值：

dict = {“Name”: "Chris", "Age": 25,}dict[’Name’]
dict[x] where x=’Name’
dict.get(‘Name’, default) or dict.get(x, default)

Web API和配置文件 (Web API and Configuration Files)

When using Python, almost all external documents are converted through one of these formats into dictionaries: JSON/YAML configuration files, messages exchanged via Web API, or AWS lambda events. XML sometimes is also used.

使用Python时，几乎所有外部文档都通过以下格式之一转换为字典：JSON / YAML配置文件，通过Web API交换的消息或AWS lambda事件。有时也使用XML。

AWS开发工具包 (AWS SDK)

Our team often has to work with deeply nested files like data coming from the AWS SDK or as an event parameter of the Lambda function handler.

我们的团队经常必须处理深度嵌套的文件，例如来自AWS开发工具包的数据或作为Lambda函数处理程序的事件参数。

{
  'Buckets': [{
      'Name': 'string',
      'CreationDate': datetime(2015, 1, 1)
  }],
  'Owner': {
  'DisplayName': 'string',
  'ID': 'string'
  }
}

代码读写速度优化 (Code Write/Read Speed Optimization)

The problem is work efficiency. For example, JavaScript notation requires only 75% (one dot character vs two brackets and quotes) of the writing and reading overhead when compared to Python.

问题是工作效率。例如，与Python相比，JavaScript表示法仅需要75％(一个点字符，两个方括号和引号)的读写开销。

属性访问 (Attribute Access)

In order to provide non-trivial access to attributes in Python, one has to implement two magic methods: __getattr__ and __setattr __.

为了提供对Python属性的非平凡访问，必须实现两种魔术方法： __getattr__和__setattr __ 。

Based on the discussion above, we need to extend the behavior of the existing dict class with these two magic methods. The adapter design pattern accomplishes this task. There are two options to consider: Object Adapter or Class Adapter.

基于以上讨论，我们需要使用这两种魔术方法扩展现有dict类的行为。适配器设计模式可以完成此任务。要考虑两个选项：对象适配器或类适配器。

评估对象适配器 (Evaluating Object Adapter)

Applying the Object Adapter design pattern means wrapping the original dict object with an external one and implementing the required magic methods.

应用对象适配器设计模式意味着将原始dict对象与外部对象包装在一起并实现所需的魔术方法。

Python collections.abc (Python collections.abc)

One possibility is to implement Mapping and Mutable Mapping abstractions from the collections.abc module, then to add __getattr__ and __setattr__ magic methods to them. Indeed, that was how the initial version of jdict was implemented.

一种可能性是从collections.abc模块中实现Mapping和Mutable Mapping抽象，然后向其中添加__getattr__和__setattr__魔术方法。确实，这就是jdict最初版本的实现方式。

This method turned out to be heavyweight and inefficient:

结果证明这种方法是重量级且效率低下的：

It required reproducing all the methods of the Python dictionary.
它需要重现Python字典的所有方法。
It behaved even worse when we needed to deal with deeply nested data structures. To learn how we finally addressed nested structures, see the JSON hook and botocore patch sections below.
当我们需要处理深度嵌套的数据结构时，它的表现甚至更糟。要了解我们最终如何解决嵌套结构，请参阅下面的JSON钩子和botocore补丁程序部分。

UserDict (UserDict)

UserDict is another possible form of Object Adapter for a Python dictionary. In this case, it comes from the Python standard library.

UserDict是Python字典的对象适配器的另一种可能形式。在这种情况下，它来自Python标准库。

Using this option does not offer any significant advantage, since:

使用此选项没有任何明显的优势，因为：

After Python 2.2, it’s possible to inherit directly from the built-in dict class.
在Python 2.2之后，可以直接从内置的dict类继承。
We also have to reproduce the magic methods of the attribute.
我们还必须重现该属性的魔术方法。
It incurs the overhead of regular __getitem__, __setitem__ operations.
这会产生常规__getitem __，__ setitem__操作的开销。

命名元组 (Named Tuples)

Another idea was to make the dictionary behave like named tuples, which supports attribute-level access.

另一个想法是使字典的行为像命名元组，它支持属性级访问。

This approach also turned out to be ineffective:

这种方法也被证明是无效的：

It created a complete copy of original dictionary and thus was impractical from a performance point of view.
它创建了原始词典的完整副本，因此从性能角度来看是不切实际的。
It did not solve the nested data structure problem.
它没有解决嵌套数据结构问题。

Jdict类适配器 (Jdict Class Adapter)

After completing our the research, we came to the conclusion that applying the Class Adapter design pattern has the best potential.

完成研究后，我们得出的结论是，应用类适配器设计模式具有最大的潜力。

The class adapter uses inheritance and can only extend the base class and supply additional functionality to it.

类适配器使用继承，并且只能扩展基类并为其提供附加功能。

This is how our Class Adapter code looks:

这是我们的类适配器代码的外观：

from typing import Any, Union
from copy import deepcopy
import jsonclass jdict(dict):
   """
   The class gives access to the dictionary through the attribute  name.
   """   def __getattr__(self, name: str) -> Union[Any]:
       try:
           return self.__getitem__(name)
       except KeyError:
           raise AttributeError(name + ' not in dict')   def __setattr__(self, key: str, value: Any) -> None:
       self.__setitem__(key, value)

深复制 (deepcopy)

def __deepcopy__(self, memo):
    return jdict((k, deepcopy(v, memo)) for k,v in self.items())

We also added the __deepcopy__ method to the adapter. Without this magic method deepcopy() a jdict object will produce a dict object, thus losing the advantage of attribute-level access.

我们还向适配器添加了__deepcopy__方法。没有这个魔术方法deepcopy()，一个jdict对象将产生一个dict对象，从而失去了属性级访问的优势。

from caios.jdict import jdict
import copypy_dict = dict(a = [1, 2, 3], b = 7)
j_dict = jdict(a = [1, 2, 3], b = 7)
py_copy = copy.deepcopy(py_dict)
j_copy = copy.deepcopy(j_dict)print(type(py_copy))
<class 'dict'>print(type(j_copy))
<class 'caios.jdict.jdict.jdict'>

处理嵌套数据结构 (Dealing with nested data structures)

While applying the Class Adapter design pattern turned out to be the optimal starting point, it still left open the question of how to deal with nested data structures. In other words, what should be done about having jdict containing another dict.

虽然应用类适配器设计模式被证明是最佳的起点，但是仍然留下了如何处理嵌套数据结构的问题。换句话说，关于包含另一个字典的判决应该怎么做。

In order to solve this problem, we need to consider separately JSON object deserialization and explicit creation of a dict somewhere in the underlying SDK.

为了解决此问题，我们需要分别考虑JSON对象反序列化和在基础SDK中某个位置显式创建dict。

JSON解码 (JSON Decoding)

When working with data that we receive from external sources in JSON format, the following translations are performed by default when decoding in python:

当使用从外部源以JSON格式接收的数据时，在python中解码时，默认情况下执行以下转换：

Image for post — Fig.2: Json decoding in python

An object_pairs_hook, if specified, will be called with the result of every JSON object decoded with an ordered list of pairs. The return value of object_pairs_hook will be used instead of the dict. This feature can be used to implement custom decoders. If object_hook also is defined, then the object_pairs_hook takes priority.

如果指定了object_pairs_hook ，则将对每个JSON对象的结果进行解码，并按对的有序列表进行解码。将使用object_pairs_hook的返回值代替dict 。此功能可用于实现自定义解码器。如果还定义了object_hook，则object_pairs_hook优先。

Thus, we utilize this hook in order to create jdict instead of dict during JSON decoding, This approach covers 80% of the cases we practically have to deal with.

因此，我们利用该钩子在JSON解码期间创建jdict而不是dict。这种方法涵盖了我们实际上必须处理的80％的情况。

json._default_decoder = json.JSONDecoder(object_pairs_hook=jdict)

Botocore补丁 (Botocore Patch)

The object pairs hook mentioned above, however, does not help with Boto3 SDK. The reason for this is that AWS service APIs return XML instead of JSON, and the results are parsed by the BaseXMLResponseParser, which creates and populates the dict object directly.

但是，上面提到的对象对挂钩对Boto3 SDK没有帮助。这样做的原因是，AWS服务API返回XML而不是JSON，结果由BaseXMLResponseParser解析，后者直接创建并填充dict对象。

Structure of Python

Python的结构

Since in this case the JSON hook does not help, we need to look at automatic rewriting of compiled Python code.

由于在这种情况下，JSON挂钩无济于事，因此我们需要查看编译后的Python代码的自动重写。

To understand how Python works and how we can solve this problem, let’s look at the full path of the program from source code to execution.

为了了解Python的工作原理以及如何解决这个问题，让我们看一下从源代码到执行程序的完整路径。

Abstract Syntax Tree (AST)

抽象语法树(AST)

To solve the problem, based on the structure of the full path of the program from source code to execution, we need to replace the code inside the AST. By traversing the AST, we will change the regular dictionary to jdict. Thus, the Boto3 SDK will return the jdict, as is required.

为了解决该问题，基于程序从源代码到执行的完整路径的结构，我们需要替换AST内部的代码。通过遍历AST，我们将常规字典更改为jdict。因此，Boto3 SDK将根据需要返回判决。

Below is the code of the class that walks through the abstract syntax tree and changes the Python dictionary to jdict.

下面是遍历抽象语法树并将Python字典更改为jdict的类的代码。

import ast
from typing import Anyclass jdictTransformer(ast.NodeTransformer):   """
   The visitor class of the node that traverses the abstract syntax    tree and calls the visitor function for each node found. Inherits   from class NodeTransformer.
   """   def visit_Module(self, node: Any) -> Any:
       node = self.generic_visit(node)
       import_node = ast.ImportFrom(module='caios.jdict.jdict',
                                    names=[ast.alias(name='jdict')],
                                    level=0)
       node.body.insert(0, import_node)
       return node   def visit_Dict(self, node: Any) -> Any:
       node = self.generic_visit(node)
       name_node = ast.Name(id='jdict', ctx=ast.Load())
       new_node = ast.Call(func=name_node, args=[node], keywords=[])
       return new_node

Patch Module

补丁模块

Using AST, we created a patch for the module botocore. To convert XML to jdict in runtime:

使用AST，我们为模块botocore创建了一个补丁。要在运行时将XML转换为jdict：

def patch_module(module: str) -> None:
    parsers = sys.modules[module]
    filename = parsers.__dict__[‘__file__’]
    src = open(filename).read()
    inlined = transform(src)
    code = compile(inlined, filename, ‘exec’)
    exec(code, vars(parsers))

In this case, we are patching the botocore parsers file.

在这种情况下，我们正在修补botocore解析器文件。

import boto3
import caios.jdictcaios.jdict.patch_module(‘botocore.parsesrs’)

方法的局限性 (Limitations of the method)

There are several limitations to the method above:

上面的方法有一些限制：

Each Jdict instance actually stores 2 dictionaries, one inherited and another one in __dict__.
每个Jdict实例实际上都存储2个字典，一个字典是继承的，另一个在__dict__中。
If a dictionary key is not a valid Python name, attribute-based access won’t work. Consider, for example, dict.created-at. It could be either dict[‘created-at’] or dict.created_at (would require a schema change).
如果字典键不是有效的Python名称，则基于属性的访问将无法进行。考虑例如dict.created-at。可以是dict ['created-at']或dict.created_at(需要更改架构)。
Another limitation is encountered when a field name is a value of another variable. One could write dict[x] but not dict.x because dict.x means dict[‘x’], not the value of the x variable.
当字段名称是另一个变量的值时，会遇到另一个限制。可以写dict [x]但不能写dict.x，因为dict.x表示dict ['x']，而不是x变量的值。
If a dictionary contains a key of the dict class methods (e.g. keys), then accessing it via the dot notation will return the dict method, while accessing via __getitem__ will return the dictionary value. In other words d.keys will be not equal to d [‘keys’]?
如果字典包含dict类方法的键(例如键)，则通过点表示法访问它将返回dict方法，而通过__getitem__访问将返回字典值。换句话说，d.keys将不等于d ['keys']？

追求 (To Be Pursued)

At the moment, our program does not use such configuration files as YAML (we don’t need them at the moment). Also, the program does not support csv and tables. We are currently in the development of a program that will work with AWS tables.

目前，我们的程序未使用YAML这样的配置文件(目前不需要)。此外，该程序不支持csv和表。我们目前正在开发一个可与AWS表一起使用的程序。

第三方图书馆 (Third Party Libraries)

While working on this project, we did not discover any suitable third-party libraries to utilize . At the time of final writing for this article, I did, in fact, encounter several possibilities, namely:

在进行此项目时，我们没有发现要使用的任何合适的第三方库。在撰写本文的最后一篇文章时，实际上，我确实遇到了几种可能性，即：

In our project, we conceivably could use any of these options. All three are based on the idea of creating an adapter and overriding the dictionary functions in it. Plus, some of them add functionality that is not required for our work.

可以想象，在我们的项目中，可以使用任何这些选项。这三个都是基于创建适配器并覆盖其中的字典功能的想法。另外，其中一些添加了我们工作不需要的功能。

结论 (Conclusions)

This effort demonstrates that dictionary objects can be managed nearly as efficiently in Python as in JavaScript. Our team believes it noticeably will increase productivity.
这项工作表明，字典对象在Python中的管理几乎与在JavaScript中一样有效。我们的团队认为，这将明显提高生产率。
This effort simplified the effort using Python dictionaries. Now, we do not need to “break fingers” by typing endless sequences of brackets and quotes.
这项工作简化了使用Python词典的工作。现在，我们不需要通过无休止的括号和引号序列来“打断手指”。
The JSON hook solved the nested data structures problem for JSON decoding This covers 80% of the cases we practically encounter.
JSON挂钩解决了JSON解码的嵌套数据结构问题。这涵盖了我们实际遇到的80％的情况。
The botcore patch solved the problem of results coming from Boto3 SDK which parses AWS services API results arriving in XML, and builds dict objects on the spot. If required, the same patch could be applied to other libraries.
该botcore补丁解决了Boto3 SDK的结果问题，该Boto3 SDK解析以XML格式到达的AWS服务API结果，并当场构建dict对象。如果需要，可以将相同的补丁程序应用于其他库。