python 64式: 第25式、python库之i18n原理分析

本文链接：https://blog.csdn.net/qingyuanluofeng/article/details/90348682

本片介绍i18n的源码，探究国际化翻译的原理。
主要分为如下内容:
1 i18n中的两种使用方式
2 i18n的懒加载模式
3 i18n的源码分析
4 总结

1 i18n中的两种使用方式
i18n是用来进行国际化翻译的。经过调查，现在主要有两种翻译方式。
1) 直接通过gettext方法，显示调用
实例: horizon的国际化:
scope.operationlogi18n = {
'Create Instance': gettext('Create Instance'),
'Shutdown Instance': gettext('Shutdown Instance'),
}
分析:
直接调用gettext方法，获取翻译结果

2) 通过_方法使用懒汉加载模式
_方法实际是:
import oslo_i18n
# ref: https://docs.openstack.org/oslo.i18n/ocata/usage.html
DOMAIN = "myproject"
_translators = oslo_i18n.TranslatorFactory(domain=DOMAIN)
_ = _translators.primary

这种方式具体参见后续源码分析。

2 i18n的懒加载模式
Lazy Translation¶
Lazy translation delays converting a message string to the translated form as long as possible, including possibly never if the message is not logged or delivered to the user in some other way. It also supports logging translated messages in multiple languages, by configuring separate log handlers.

Lazy translation is implemented by returning a special object from the translation function, instead of a unicode string. That special message object supports some, but not all, string manipulation APIs. For example, concatenation with addition is not supported, but interpolation of variables is supported. Depending on how translated strings are used in an application, these restrictions may mean that lazy translation cannot be used, and so it is not enabled by default.

To enable lazy translation, call enable_lazy().

import oslo_i18n

oslo_i18n.enable_lazy()

全局搜索aodh代码:
enable_lazy
最终发现在:
aodh/service.py中有

def prepare_service(argv=None, config_files=None):
conf = cfg.ConfigOpts()
oslo_i18n.enable_lazy()
log.register_options(conf)
log_levels = (conf.default_log_levels +
['futurist=INFO', 'keystoneclient=INFO'])
log.set_defaults(default_log_levels=log_levels)
defaults.set_cors_middleware_defaults()
db_options.set_defaults(conf)
policy_opts.set_defaults(conf)
from aodh import opts
# Register our own Aodh options
for group, options in opts.list_opts():
conf.register_opts(list(options),
group=None if group == "DEFAULT" else group)
keystone_client.register_keystoneauth_opts(conf)

conf(argv, project='aodh', validate_default_values=True,
default_config_files=config_files)

ka_loading.load_auth_from_conf_options(conf, "service_credentials")
log.setup(conf, 'aodh')
messaging.setup()
return conf

里面有一行内容:
oslo_i18n.enable_lazy()

该prepare_service方法被app.py中的_app方法使用，也被build_wsgi_app方法使用。

参考:
https://docs.openstack.org/oslo.i18n/latest/user/usage.html

3 i18n的源码分析
3.0 总入口
自己编写的i18n.py文件
import oslo_i18n

# ref: https://docs.openstack.org/oslo.i18n/ocata/usage.html

DOMAIN = "myproject"

_translators = oslo_i18n.TranslatorFactory(domain=DOMAIN)

_ = _translators.primary

_LI = _translators.log_info
_LW = _translators.log_warning
_LE = _translators.log_eror
_LC = _translators.log_critical

def get_available_languages():
return oslo_i18n.get_available_languages(DOMAIN)

def translate(value, user_locale):
return oslo_i18n.translate(value, user_locale)
分析:
1）上述先设置了_方法
2）假设有需要使用国际化翻译的地方，可以使用如下形式进行国际化
引用_，_LI等
from i18n import _, _LI
LOG.info(_LI('enter app_factory'))
或者这种用法，推荐用_而不是_LI
result = _('enter app_factory')
LOG.info(result)

3） _方法内部调用了_make_translation_func方法，内容如下

def _make_translation_func(self, domain=None):
"""Return a translation function ready for use with messages.
The returned function takes a single value, the unicode string
to be translated. The return type varies depending on whether
lazy translation is being done. When lazy translation is
enabled, :class:`Message` objects are returned instead of
regular :class:`unicode` strings.
The domain argument can be specified to override the default
from the factory, but the localedir from the factory is always
used because we assume the log-level translation catalogs are
installed in the same directory as the main application
catalog.
"""
if domain is None:
domain = self.domain
t = gettext.translation(domain,
localedir=self.localedir,
fallback=True)
# Use the appropriate method of the translation object based
# on the python version.
m = t.gettext if six.PY3 else t.ugettext

def f(msg):
"""oslo_i18n.gettextutils translation function."""
if _lazy.USE_LAZY:
return _message.Message(msg, domain=domain)
return m(msg)
return f

3.1 总入口
103 if not desired_locale:
104 system_locale = locale.getdefaultlocale()

(Pdb) p system_locale
('zh_CN', 'UTF-8')

111 ->   locale_dir = os.environ.get(
112     _locale.get_locale_dir_variable_name(domain)
113     )
114     lang = gettext.translation(domain,
115     localedir=locale_dir,
116     languages=[desired_locale],
117     fallback=True)
118
119     if not has_contextual_form and not has_plural_form:
120     # This is the most common case, so check it first.
121     translator = lang.gettext if six.PY3 else lang.ugettext
122     translated_message = translator(msgid)
123
124     elif has_contextual_form and has_plural_form:
125     # Reserved for contextual and plural translation function,
126     # which is not yet implemented.
127     raise ValueError("Unimplemented.")

> /usr/lib/python2.7/site-packages/oslo_i18n/_locale.py(25)get_locale_dir_variable_name()->'MYPROJECT_LOCALEDIR'
-> return domain.upper().replace('.', '_').replace('-', '_') + '_LOCALEDIR'

分析:
111     locale_dir = os.environ.get(
112 ->   _locale.get_locale_dir_variable_name(domain)
113     )

果然是获取环境变量做的。

(Pdb) p desired_locale
'zh_CN'

获取了语言:
(Pdb) p lang
<gettext.GNUTranslations instance at 0x7f8ad9e3e200>

119     if not has_contextual_form and not has_plural_form:
120     # This is the most common case, so check it first.
121 ->   translator = lang.gettext if six.PY3 else lang.ugettext
122     translated_message = translator(msgid)
123
124     elif has_contextual_form and has_plural_form:
125     # Reserved for contextual and plural translation function,
126     # which is not yet implemented.

(Pdb) p translated_message
u'\u8fdb\u5165app\u5de5\u5382'

翻译之后有:
(Pdb) p info
u'\u8fdb\u5165app\u5de5\u5382'

总结:
经过调试发现，所谓的oslo_i18n的懒加载模式实际是获取localedir,获取当前环境变量中的
语言，然后找到对应的翻译方法，对msgid进行翻译，获取对应的msgstr，因而能翻译出对应的语言。

使用不使用懒加载模式，按照道理应该不影响。

继续调试aodh中，查看为什么翻译失败，是否是因为其他原因。

485 ->   try:
486     # check if it's supported by the _locale module
487     import _locale
488     code, encoding = _locale._getdefaultlocale()

AttributeError: "'module' object has no attribute '_getdefaultlocale'"
(Pdb) p _locale
<module '_locale' from '/usr/lib64/python2.7/lib-dynload/_localemodule.so'>

3.2 环境变量
(Pdb) p envvars
('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')

501     import os
502     lookup = os.environ.get
503     for variable in envvars:
504     localename = lookup(variable,None)
505 ->   if localename:
506     if variable == 'LANGUAGE':
507     localename = localename.split(':')[0]
508     break
509     else:
510     localename = 'C'

(Pdb) p variable
'LC_ALL'
(Pdb) p localename
'en_US.utf8'

> /usr/lib64/python2.7/locale.py(505)getdefaultlocale()
-> if localename:
(Pdb) p localename
None
(Pdb) n
> /usr/lib64/python2.7/locale.py(503)getdefaultlocale()
-> for variable in envvars:

分析:
终于知道了
461 -> def getdefaultlocale(envvars=('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')):
这里有一个默认搜索的先后顺序，
在myproject中这个是LC_ALL为空，而在其他地方不为空。

3.3 完整代码
01     import os
502     lookup = os.environ.get
503     for variable in envvars:
504     localename = lookup(variable,None)
505     if localename:
506     if variable == 'LANGUAGE':
507     localename = localename.split(':')[0]
508     break
509     else:
(Pdb) l
510     localename = 'C'
511     return _parse_localename(localename)

LC_CTYPE这个也为空。

(Pdb) p envvars
('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')

(Pdb) p localename
'zh_CN.UTF-8'
(Pdb) p variable
'LANG'

最终进入到这里:
总结:
这里整体的逻辑是在寻找语言编码的时候默认是按照这个顺序:
('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')
去查找对应环境变量，一旦找到环境变量，就对该环境变量解析，

然后调用下面的方法:
> /usr/lib64/python2.7/locale.py(415)_parse_localename()

3.4 分析_parse_localename方法
415 ->   def _parse_localename(localename):
416
417     """ Parses the locale code for localename and returns the
418     result as tuple (language code, encoding).
419
420     The localename is normalized and passed through the locale
421     alias engine. A ValueError is raised in case the locale name
422     cannot be parsed.
423
424     The language code corresponds to RFC 1766. code and encoding
425     can be None in case the values cannot be determined or are
426     unknown to this implementation.
427
428     """
429     code = normalize(localename)
430     if '@' in code:
431     # Deal with locale modifiers
432     code, modifier = code.split('@')
433     if modifier == 'euro' and '.' not in code:
434     # Assume Latin-9 for @euro locales. This is bogus,
435     # since some systems may use other encodings for these
436     # locales. Also, we ignore other modifiers.
437     return code, 'iso-8859-15'
438
439     if '.' in code:
440     return tuple(code.split('.')[:2])
441     elif code == 'C':
442     return None, None
443     raise ValueError, 'unknown locale: %s' % localename

分析:
(Pdb) p code
'zh_CN.UTF-8'

439 -> if '.' in code:
440 return tuple(code.split('.')[:2])

按点号分割得到:
('zh_CN', 'UTF-8')

3.5 继续分析_translate_msgid方法
最后回到:
> /usr/lib/python2.7/site-packages/oslo_i18n/_message.py(106)_translate_msgid()

101     def _translate_msgid(msgid, domain, desired_locale=None,
102     has_contextual_form=False, has_plural_form=False):
103     if not desired_locale:
104     system_locale = locale.getdefaultlocale()
105     # If the system locale is not available to the runtime use English
106 ->   if not system_locale or not system_locale[0]:
107     desired_locale = 'en_US'
108     else:
109     desired_locale = system_locale[0]
110
111     locale_dir = os.environ.get(
112     _locale.get_locale_dir_variable_name(domain)
113     )
114     lang = gettext.translation(domain,
115     localedir=locale_dir,
116     languages=[desired_locale],
117     fallback=True)
118
119     if not has_contextual_form and not has_plural_form:
120     # This is the most common case, so check it first.
121     translator = lang.gettext if six.PY3 else lang.ugettext
122     translated_message = translator(msgid)
123
124     elif has_contextual_form and has_plural_form:
125     # Reserved for contextual and plural translation function,
126     # which is not yet implemented.
127     raise ValueError("Unimplemented.")
128
129     elif has_contextual_form:
130     (msgctx, msgtxt) = msgid
131     translator = lang.gettext if six.PY3 else lang.ugettext
132
133     msg_with_ctx = "%s%s%s" % (msgctx, CONTEXT_SEPARATOR, msgtxt)
134     translated_message = translator(msg_with_ctx)
135
136     if CONTEXT_SEPARATOR in translated_message:
137     # Translation not found, use the original text
138     translated_message = msgtxt
139
140     elif has_plural_form:
141     (msgsingle, msgplural, msgcount) = msgid
142     translator = lang.ngettext if six.PY3 else lang.ungettext
143     translated_message = translator(msgsingle, msgplural, msgcount)
144
145     return translated_message

分析:
1)
109 -> desired_locale = system_locale[0
(Pdb) p desired_locale
'zh_CN

2)
然后调用下面的方法获取localedir的真实路径:
111 ->   locale_dir = os.environ.get(
112     _locale.get_locale_dir_variable_name(domain)
113     )

3)
18 ->   def get_locale_dir_variable_name(domain):
19     """Build environment variable name for local dir.
20
21     Convert a translation domain name to a variable for specifying
22     a separate locale dir.
23
24     """
25     return domain.upper().replace('.', '_').replace('-', '_') + '_LOCALEDIR'

得到输出结果如下:
'MYPROJECT_LOCALEDIR'

最终得到:
(Pdb) p locale_dir
'/home/machao/myproject/test_project/myproject/myproject/locale'

4) 获取对应语言的翻译器
114 ->   lang = gettext.translation(domain,
115     localedir=locale_dir,
116     languages=[desired_locale],
117     fallback=True)

得到如下结果:
(Pdb) p lang
<gettext.GNUTranslations instance at 0x7f2a6ff7c5a8>

5) 执行真正的翻译
119     if not has_contextual_form and not has_plural_form:
120     # This is the most common case, so check it first.
121 ->   translator = lang.gettext if six.PY3 else lang.ugettext
122     translated_message = translator(msgid)

(Pdb) p translator
<bound method GNUTranslations.ugettext of <gettext.GNUTranslations instance at 0x7f2a6ff7c5a8>>

最终调用如下方法:
> /usr/lib64/python2.7/gettext.py(403)ugettext()

3.5 分析ugettext方法
400 def ugettext(self, message):
401     missing = object()
402     tmsg = self._catalog.get(message, missing)
403     if tmsg is missing:
404     if self._fallback:
405     return self._fallback.ugettext(message)
406     return unicode(message)
407     return tmsg

分析:
1)
(Pdb) p missing
<object object at 0x7f2a80f35460>

2)
(Pdb) p message
'enter app_factory'
(Pdb) p missing
<object object at 0x7f2a80f35460>
(Pdb) p tmsg
u'\u8fdb\u5165app\u5de5\u5382'

3)
关键就是调用:
tmsg = self._catalog.get(message, missing)

这个应该就是到对应的目录下找到mo中东西翻译

4) 验证aodh中下面几个环境变量
('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')

执行命令:
env|egrep 'LC_ALL|LC_CTYPE|LANG|LANGUAGE'

样例输出结果如下:
AGE'oot@aodh-notifier-7649b98786-2dd45 /]# env|egrep 'LC_ALL|LC_CTYPE|LANG|LANGU
LC_ALL=en_US.utf8
LANG=zh_CN

因为LC_ALL的优先级高于LANG,所以导致这里国际化翻译失败。

解决方法:
同时设置这几个变量

5) 去myrpoject项目所在环境查看语言相关环境变量
(Pdb) [root@localhost myproject]# env|egrep 'LC_ALL|LC_CTYPE|LANG|LANGUAGE'
LANG=zh_CN.UTF-8
GDM_LANG=zh_CN.UTF-8

只有LANG是有效的，没有LC_ALL

3.6 重新更新aodh的charts
重新升级看是否可以。

()[root@aodh-notifier-859fbd4c94-8j7gm /]# /tmp/aodh-notifier.sh
+ export AODH_LOCALEDIR=/usr/lib/python2.7/site-packages/aodh/locale
+ AODH_LOCALEDIR=/usr/lib/python2.7/site-packages/aodh/locale
+ export LC_ALL=zh_CN
+ LC_ALL=zh_CN
/tmp/aodh-notifier.sh: line 8: warning: setlocale: LC_ALL: cannot change locale (zh_CN): No such file or directory
/tmp/aodh-notifier.sh: line 8: warning: setlocale: LC_ALL: cannot change locale (zh_CN): No such file or directory
+ export LANG=zh_CN
+ LANG=zh_CN
+ exec aodh-notifier --config-file=/etc/aodh/aodh.conf
sh: warning: setlocale: LC_ALL: cannot change locale (zh_CN): No such file or directory
sh: warning: setlocale: LC_ALL: cannot change locale (zh_CN): No such file or directory
sh: warning: setlocale: LC_ALL: cannot change locale (zh_CN): No such file or directory

分析:
实际测试已经可以中英文切换

4 总结
i18n的懒加载翻译模式中通过使用oslo_i18n.TranslatorFactory(domain=DOMAIN).primary方法进行翻译，里面调用
_make_translation_func方法返回i18n自己定义Message对象，该对象包含了待翻译的信息，最后调用
_translate_msgid获取msgid对应的国际化翻译内容msgstr。
其中需要设置两样东西:
一个是项目本身的localedir目录，为的是找到对应的mo,po等文件信息；
一个是设置语言环境变量，优先级从高到低如下: 'LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE'。
设置LC_ALL即可不用设置其他环境变量。
另外注意: 设置不同语言环境变量，需要重启组件的服务才可以生效。

参考:
[1] https://docs.openstack.org/oslo.i18n/latest/user/usage.html
[2] https://blog.csdn.net/Bill_Xiang_/article/details/78570404