Python は、辞書を用いてローカル変数やグローバル変数へアクセスするための 二つのビルトイン関数、locals と globals を持っている。
locals を覚えてるだろうか？ はじめて見たのは以下のコードにおいてあろう：
def unknown_starttag(self, tag, attrs):
strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs])
self.pieces.append("<%(tag)s%(strattrs)s>" % locals())
いや、待ってほしい、読者はまだ locals について学習することはできない。 まず最初に、名前空間について、学ぶ必要があるのだ。 これは無味乾燥なものかもしれないが、重要なので注意を払ってほしい。
Python は、変数を追跡し続けるために、名前空間と呼ばれるものを用いている。 名前空間は、単に辞書のようなものであり、そのキーは「変数名」、値は「変数の値」である。 実際、すぐ確認できるのだが、Python の辞書と同じように名前空間にアクセスすることができる。
Python プログラムのいかなるポイントにおいても、いくつかの名前空間の変数が存在する。 それぞれの関数は それ自身の名前空間を持っており、それが（引数やローカル変数といった）関数の変数を追跡し続ける。 それぞれのモジュールは、「グローバルな名前空間」と呼ばれる自分自身の名前空間をもっており、それによって、（関数、クラス、他のすべでのインポートされたモジュール、モジュール・レベルの変数、定数といった）モジュール変数を追跡し続ける。 そして、ビルトイン関数や例外を保持する「ビルトイン名前空間」が存在する。
コードのある行が変数 x の値を問い合わせたとき、Python は その時点でアクセス可能なすべての名前空間の中を順番に、その変数を検索する：
1. ローカルな名前空間 --- 現在の関数やクラス・メソッドに特有な名前空間のこと。 もしも関数がローカル変数 x を定義したり、引数 x を持っていると、Python はこれを使い、検索をストップする。
2. グローバルな名前空間 --- 現在のモジュールに特有な名前空間のこと。 もしもあるモジュールがx という変数、 xという関数、あるいは、x というクラスを定義していた場合、Python はこれを用い、検索をストップする。
3. ビルトイン名前空間 --- すべてのモジュールに対してグローバルな名前空間のこと。 最後の手段として、Python は、x がビルトイン関数か変数ではないかと推測する。
もしも Python が x をいかなる名前空間においても発見出来なかった場合、ギブアップし、There is no variable named 'x' というメッセージとともに NameError を返す。 この挙動は、例3.18で見たとおりである。 読者は、解放された変数を参照する際、Python が このエラーを返す前にどれだけ多くの仕事をこなしたかについて、感謝してはこなかったであろう。
Python 2.2 は、名前空間の検索に影響するような、微妙ではあるが重要な変更を行った： それは、ネストされたスコープである。 Python 2.2 以前のバージョンでは、ネストされた関数やlambda 関数の中の変数を参照する際、Python はまず現在の（ネストされた、あるいは、lambda）関数の名前空間を検索し、次に、親の関数の名前空間を検索し、それから、モジュールの名前空間を検索していた。 Python 2.1 は、どちらの方法でも動作する； デフォルトでは、Python 2.0 のように動作するが、コードの先頭に次の行を付け加えることにより、そのモジュールを、Python 2.2 のように動作させることも可能である：
from __future__ import nested_scopes
まだ混乱しているだろうか？ おちこむことはない！ 約束するが、これは非常にクールなことである。 Python における、他の多くの事柄と同じく、名前空間は、プログラムの実行時に直接アクセス可能である。 しかし、どうやって？ ローカルの名前空間へは、ビルトインされた locals 関数によって、アクセスできる。 そして、（モジュラー・レベルでの）グローバルな名前空間には、ビルトインされた globals 関数によって、アクセスできる。
Example 8.10. locals の導入
>>> def foo(arg):
... x = 1
... print locals()
...
>>> foo(7)
{'arg': 7, 'x': 1}
>>> foo('bar')
{'arg': 'bar', 'x': 1}
関数 foo は、ローカルな名前空間に二つの変数を持っている； ひとつは arg であり、その値は関数に引き渡される。 そしてもうひとつは x であり、関数の中で定義されている。
locals は 「名前／値」ペアに関する辞書を返す。この辞書のキーは、変数名（文字列型）である。したがって、foo を 7 という引数で呼び出すと、この関数の二つのローカル変数（arg(7) と x(1) ）を含む辞書を表示する。
Python は動的なタイピングをサポートしており、arg に対して簡単に文字列を渡せることを思い出してほしい； この関数（そして、locals の呼び出し）はうまく動作する。locals は、あらゆるデータ型の変数を取り扱える。
from module import と import module の違いを覚えているだろうか？ import module では、モジュールそのものがインポートされていたが、それ自身の名前空間は保持されていた。したがって、そのモジュールが含むいかなる関数やアトリビュートに対しても、アクセスの際にはモジュール名が必要であった： module.function。しかし、from module import を用いると、本当に、特定の関数とそのアトリビュートを他のモジュールからインポートし、あなた自身がもっている名前空間に付け加えることになる。もともとのモジュールを参照することなく、関数名のみで直接アクセスができるのは、これが理由である。globals 関数を用いると、ここで起きていることを実際に眺めることができる。
Example 8.11. globals の導入
BaseHTMLProcessor.py の最下部にある、以下のコード・ブロックを見てほしい：
if __name__ == "__main__":
for k, v in globals().items():
print k, "=", v
おじけづくことはない、以前、このコードを読んだことを思い出そう。 globals 関数は辞書を返し、あなたは、items メソッドとマルチ・バリアブル・アサイメントを用いて、この辞書の中身について繰り返し反復を行った。 ここで新たに導入されたのは、globals 関数のみである。
ここで、スクリプトをコマンドラインから実行すると、以下のアウトプットが得られる。 (ただし、読者がPython をインストールした環境に応じて、このアウトプットは若干異なるかもしれないことに、注意してほしい。)
c:\docbook\dip\py> python BaseHTMLProcessor.py
SGMLParser = sgmllib.SGMLParser
htmlentitydefs = <module 'htmlentitydefs' from 'C:\Python23\lib\htmlentitydefs.py'>
BaseHTMLProcessor = __main__.BaseHTMLProcessor
__name__ = __main__
... rest of output omitted for brevity...
from module import によって、SGMLParser は sgmlib からインポートされた。これは、モジュールの名前空間へ直接とりこまれたことを意味し、実際、そうなっている。
これを、import によってインポートされた htmlentitiydefs と比べている。htmlentitydefsモジュール自身は名前空間の中にあるが、htmlentitydefs の中で定義された変数 entitydefs はそうでは無い。
このモジュールは、BaseHTMLProcessor という一つのクラスを定義しているのみであり、そうなっている。ここでの値はクラスそのものであり、クラスの特定のインスタンスでは無いことに注意せよ。
if __name__ トリックについて覚えているだろうか？ モジュールを実行する際、（他のモジュールからのインポートに反して）、ビルトインの __name__ アトリビュートは 特殊な値 __main__ になる。 このモジュールは、コマンドラインからスクリプトとして実行されているので、__name__ は __main__ であり、これが、globals を表示する小さなコードが実行された理由である。
locals 関数と globals 関数を使うと、変数名を文字列として与えることにより、任意の変数の値を動的に取得することができる。 これは、関数名を文字列として与えることにより、任意の関数へアクセスすることを可能とする、getattr 関数の機能をうつしたものである。
locals と globals 関数にはもうひとつ重要な違いがあり、それが あなたに噛み付く前に学んでほしい。 それは、とにかく いつか あなたに噛み付くだろうが、すくなくともその時、既にそれを学んだことを思い出してほしい。
Example 8.12. locals はリード・オンリーであり、 globals はそうではない
def foo(arg):
x = 1
print locals()
locals()["x"] = 2
print "x=",x
z = 7
print "z=",z
foo(3)
globals()["z"] = 8
print "z=",z
foo は、引数 3 とともに呼び出されたので、これは {'arg': 3, 'x': 1} を表示するであろう。これは、何ら驚くべきことではない。
locals は辞書を返す関数であり、ここではその辞書の値を設定している。これにより、ローカル変数 x の値が 2 に変わると思うかもしれないが、しかし、そうではない。locals は、実際にローカルな名前空間を返しているわけではなく、そのコピーを返しているのである。したがって、それを変更してもローカルな名前空間の変数の値は変わらないのである。
x=2 ではなく、x=1 が表示される。
locals によってやけどした後なので、こうやっても z の値は変わらないと思うかもしれない。しかし、今度は変わるのである。Python の内部仕様の相違（これについては深追いしない。なぜなら、筆者も完全に理解しているわけでは無いので）により、globals は、名前空間のコピーではなく、名前空間そのもを返す： これは、locals とは正反対の挙動である。従って、globals が返した辞書へのあらゆる変更は、ただちにグローバル変数に影響を及ぼす。
z = 7 ではなく z = 8 が表示される。
2014年12月16日火曜日
Python高度なimport
同じフォルダにhogehoge.pyというファイルがあったら
import hogehoge
と書くことでインポートできるし、
from hogehoge import *
と書けば、本文中でhogehoge.メソッド名なんて書かずに直接メソッド名だけでメソッドを呼び出せる。asで別名とか、まあPythonのモジュールインポートのしくみでも見てもらえるといいと思う。
以下にパッケージの話が出てこないのは話したい話を単純にするための仕様です。特に「モジュールを探す範囲を拡張」なんかは普通はパッケージの仕組みを使ったほうが簡単にできると思う。でもそれだけでは出来ないことをここでは解説したい。
同じディレクトリに置きたくない！ってこともあると思います。そういうときは
sys.path.append('パス')
としてやればそこも探してくれるようになります。基本的には絶対パス。
import sys,os
sys.path.append(os.pardir) #親フォルダを追加
from i3r import printnum, printstr
とか
import sys,os
dirpath = os.path.dirname(os.path.abspath(__file__))
datadir = 'data'
sys.path.append(dirpath+'/'+datadir)
from i3k import *
とか自由にどうぞ。os.path.dirname(os.path.abspath(__file__))を使ってディレクトリの絶対パスを取得することで、ユーザによる Python コードの変更が必要となるため、sys.path.append() の使用はお勧めしません。なんて言いがかりも避けられる。
メソッドの上書き
たとえばi3r.pyで
def printnum():
print(3)
def printstr():
print("r")
と書いていて、i3k.pyで
from i3r import printnum, printstr
def printstr():
print('k')
とするとprintstr()の結果は上書きされてkになり、printnum()はそのまま3が出力される。
変数でモジュール名を指定
import式と等価な組み込み関数に__import__がある。
これを使えば、動的に生成されたモジュールを呼び出すことができ、メタプログラミングの際に有用だ。例えば下記の様になる。
modname = 'i3k'
module = __import__(modname)
module.printnum()
module.printstr()
ではfrom ... import ... はどうやって書くかというと、
modname = 'i3k'
methods = ['printnum','printstr']
module = __import__(modname, globals(), locals(), methods, -1)
module.printnum()
module.printstr()
こんな感じでimportする関数を指定できる。問題はmodule.を付けなければいけない事。さっきみたいに上書きとかしたい時はどうするか？
modname = 'i3k'
methods = ['printnum','printstr']
module = __import__(modname, globals(), locals(), methods, -1)
printnum = module.printnum #module.無しで使いたかったらコレ
printstr = getattr(module,'printstr') #この書き方でも同等
printnum()
printstr()
こんな感じでローカル変数に持って来れる。しかし、こうやって各個上書きするのは面倒だろう。for文とかでできないか？
modname = 'i3k'
methods = ['printnum','printstr']
module = __import__(modname, globals(), locals(), methods, -1)
for x in methods:
locals()[x]=getattr(module,x)
printnum()
printstr()
そう。「ローカル変数に持って来」ればいいのだからこの手が使える。ここまで来たらfrom i3k import *のようにメソッド一括みたいなこともできるかな？
modname = 'i3k'
module = __import__(modname)
exceplist = ['__builtins__', '__doc__', '__file__', '__name__', '__package__']
for x in dir(module):
if x not in exceplist:
locals()[x]=getattr(module,x)
printnum()
printstr()
可読性の面からは全然お勧めできないね。dir(モジュール名)でモジュールの全attributeを列挙してくれるから、その中から一般的に上書きしちゃいけないモノリストを挙げて、それ以外を上書きしている。まあ、あくまでこういうこともできるってことで。こういうメタなことをする時に重要なのはBuilt-in Functionsの知識。__import__もgetattrもdirも組み込み関数。
Pythonで可変関数
さっきの
locals()[x]=getattr(module,x)
ってのは前述のように「ローカル変数に持って来」てるのだが、可変関数を作っていると見ることができる。同様に、右辺に来るのをただの値にすれば可変変数も作れる。PHPは可変変数が楽だよとよく言われるが、Pythonも難しくないです。
ちなみに「可変変数」って変だなあと思ったが、英語でも"Variable variables"らしくて"Dynamic variables"とも言うが「動的変数」って別物について使うらしいし、召喚変数とかどうだろう。
import hogehoge
と書くことでインポートできるし、
from hogehoge import *
と書けば、本文中でhogehoge.メソッド名なんて書かずに直接メソッド名だけでメソッドを呼び出せる。asで別名とか、まあPythonのモジュールインポートのしくみでも見てもらえるといいと思う。
以下にパッケージの話が出てこないのは話したい話を単純にするための仕様です。特に「モジュールを探す範囲を拡張」なんかは普通はパッケージの仕組みを使ったほうが簡単にできると思う。でもそれだけでは出来ないことをここでは解説したい。
同じディレクトリに置きたくない！ってこともあると思います。そういうときは
sys.path.append('パス')
としてやればそこも探してくれるようになります。基本的には絶対パス。
import sys,os
sys.path.append(os.pardir) #親フォルダを追加
from i3r import printnum, printstr
とか
import sys,os
dirpath = os.path.dirname(os.path.abspath(__file__))
datadir = 'data'
sys.path.append(dirpath+'/'+datadir)
from i3k import *
とか自由にどうぞ。os.path.dirname(os.path.abspath(__file__))を使ってディレクトリの絶対パスを取得することで、ユーザによる Python コードの変更が必要となるため、sys.path.append() の使用はお勧めしません。なんて言いがかりも避けられる。
メソッドの上書き
たとえばi3r.pyで
def printnum():
print(3)
def printstr():
print("r")
と書いていて、i3k.pyで
from i3r import printnum, printstr
def printstr():
print('k')
とするとprintstr()の結果は上書きされてkになり、printnum()はそのまま3が出力される。
変数でモジュール名を指定
import式と等価な組み込み関数に__import__がある。
これを使えば、動的に生成されたモジュールを呼び出すことができ、メタプログラミングの際に有用だ。例えば下記の様になる。
modname = 'i3k'
module = __import__(modname)
module.printnum()
module.printstr()
ではfrom ... import ... はどうやって書くかというと、
modname = 'i3k'
methods = ['printnum','printstr']
module = __import__(modname, globals(), locals(), methods, -1)
module.printnum()
module.printstr()
こんな感じでimportする関数を指定できる。問題はmodule.を付けなければいけない事。さっきみたいに上書きとかしたい時はどうするか？
modname = 'i3k'
methods = ['printnum','printstr']
module = __import__(modname, globals(), locals(), methods, -1)
printnum = module.printnum #module.無しで使いたかったらコレ
printstr = getattr(module,'printstr') #この書き方でも同等
printnum()
printstr()
こんな感じでローカル変数に持って来れる。しかし、こうやって各個上書きするのは面倒だろう。for文とかでできないか？
modname = 'i3k'
methods = ['printnum','printstr']
module = __import__(modname, globals(), locals(), methods, -1)
for x in methods:
locals()[x]=getattr(module,x)
printnum()
printstr()
そう。「ローカル変数に持って来」ればいいのだからこの手が使える。ここまで来たらfrom i3k import *のようにメソッド一括みたいなこともできるかな？
modname = 'i3k'
module = __import__(modname)
exceplist = ['__builtins__', '__doc__', '__file__', '__name__', '__package__']
for x in dir(module):
if x not in exceplist:
locals()[x]=getattr(module,x)
printnum()
printstr()
可読性の面からは全然お勧めできないね。dir(モジュール名)でモジュールの全attributeを列挙してくれるから、その中から一般的に上書きしちゃいけないモノリストを挙げて、それ以外を上書きしている。まあ、あくまでこういうこともできるってことで。こういうメタなことをする時に重要なのはBuilt-in Functionsの知識。__import__もgetattrもdirも組み込み関数。
Pythonで可変関数
さっきの
locals()[x]=getattr(module,x)
ってのは前述のように「ローカル変数に持って来」てるのだが、可変関数を作っていると見ることができる。同様に、右辺に来るのをただの値にすれば可変変数も作れる。PHPは可変変数が楽だよとよく言われるが、Pythonも難しくないです。
ちなみに「可変変数」って変だなあと思ったが、英語でも"Variable variables"らしくて"Dynamic variables"とも言うが「動的変数」って別物について使うらしいし、召喚変数とかどうだろう。
2014年12月15日月曜日
Metaprogramming
Objects are created by other objects: special objects called "classes" that we can set up to spit out objects that are configured to our liking.
Classes are just objects, and they can be modified the same way:
>>> class Foo: pass
...
>>> Foo.field = 42
>>> x = Foo()
>>> x.field
42
>>> Foo.field2 = 99
>>> x.field2
99
>>> Foo.method = lambda self: "Hi!"
>>> x.method()
'Hi!'
To modify a class, you perform operations on it like any other object. You can add and subtract fields and methods, for example. The difference is that any change you make to a class affects all the objects of that class, even the ones that have already been instantiated.
What creates these special "class" objects? Other special objects, called metaclasses.
The default metaclass is called type and in the vast majority of cases it does the right thing. In some situations, however, you can gain leverage by modifying the way that classes are produced ? typically by performing extra actions or injecting code. When this is the case, you can use metaclass programming to modify the way that some of your class objects are created.
It's worth re-emphasizing that in the vast majority of cases, you don't need metaclasses, because it's a fascinating toy and the temptation to use it everywhere can be overwhelming. Some of the examples in this chapter will show both metaclass and non-metaclass solutions to a problem, so you can see that there's usually another (often simpler) approach.
Some of the functionality that was previously only available with metaclasses is now available in a simpler form using class decorators. It is still useful, however, to understand metaclasses, and certain results can still be achieved only through metaclass programming.
Basic Metaprogramming
So metaclasses create classes, and classes create instances. Normally when we write a class, the default metaclass type is automatically invoked to create that class, and we aren't even aware that it's happening.
It's possible to explicitly code the metaclass' creation of a class. type called with one argument produces the type information of an existing class; type called with three arguments creates a new class object. The arguments when invoking type are the name of the class, a list of base classes, and a dictionary giving the namespace for the class (all the fields and methods). So the equivalent of:
class C: pass
is:
C = type('C', (), {})
Classes are often referred to as "types," so this reads fairly sensibly: you're calling a function that creates a new type based on its arguments.
We can also add base classes, fields and methods:
# Metaprogramming/MyList.py
def howdy(self, you):
print("Howdy, " + you)
MyList = type('MyList', (list,), dict(x=42, howdy=howdy))
ml = MyList()
ml.append("Camembert")
print(ml)
print(ml.x)
ml.howdy("John")
print(ml.__class__.__class__)
""" Output:
['Camembert']
42
Howdy, John
"""
Note that printing the class of the class produces the metaclass.
The ability to generate classes programmatically using type opens up some interesting possibilities. Consider the GreenHouseLanguage.py example in the Jython chapter ? all the subclasses in that case were written using repetetive code. We can automate the generation of the subclasses using type:
# Metaprogramming/GreenHouse.py
class Event(object):
events = [] # static
def __init__(self, action, time):
self.action = action
self.time = time
Event.events.append(self)
def __cmp__ (self, other):
"So sort() will compare only on time."
return cmp(self.time, other.time)
def run(self):
print("%.2f: %s" % (self.time, self.action))
@staticmethod
def run_events():
Event.events.sort();
for e in Event.events:
e.run()
def create_mc(description):
"Create subclass using the 'type' metaclass"
class_name = "".join(x.capitalize() for x in description.split())
def __init__(self, time):
Event.__init__(self, description + " [mc]", time)
globals()[class_name] = \
type(class_name, (Event,), dict(__init__ = __init__))
def create_exec(description):
"Create subclass by exec-ing a string"
class_name = "".join(x.capitalize() for x in description.split())
klass = """
class %s(Event):
def __init__(self, time):
Event.__init__(self, "%s [exec]", time)
""" % (class_name, description)
exec klass in globals()
if __name__ == "__main__":
descriptions = ["Light on", "Light off", "Water on", "Water off",
"Thermostat night", "Thermostat day", "Ring bell"]
initializations = "ThermostatNight(5.00); LightOff(2.00); \
WaterOn(3.30); WaterOff(4.45); LightOn(1.00); \
RingBell(7.00); ThermostatDay(6.00)"
[create_mc(dsc) for dsc in descriptions]
exec initializations in globals()
[create_exec(dsc) for dsc in descriptions]
exec initializations in globals()
Event.run_events()
""" Output:
1.00: Light on [mc]
1.00: Light on [exec]
2.00: Light off [mc]
2.00: Light off [exec]
3.30: Water on [mc]
3.30: Water on [exec]
4.45: Water off [mc]
4.45: Water off [exec]
5.00: Thermostat night [mc]
5.00: Thermostat night [exec]
6.00: Thermostat day [mc]
6.00: Thermostat day [exec]
7.00: Ring bell [mc]
7.00: Ring bell [exec]
"""
The Event base class is the same. The classes are created automatically using the create_mc() function, which takes its description argument and generates a class name from it. Then it defines an __init__() method, which it puts into the namespace dictionary for the type call, producing a new subclass of Event. Note that the resulting class must be inserted into the global namespace, otherwise it will not be seen.
This approach works fine, but then consider the subsequent create_exec() function, which accomplishes the same thing by calling exec on a string defining the class. This will be much easier to understand by the vast majority of the people reading your code: those who do not understand metaclasses.
The Metaclass Hook
So far, we've only used the type metaclass directly. Metaclass programming involves hooking our own operations into the creation of class objects. This is accomplished by:
Writing a subclass of the metaclass type.
Inserting the new metaclass into the class creation process using the metaclass hook.
In Python 2.x, the metaclass hook is a static field in the class called __metaclass__. In the ordinary case, this is not assigned so Python just uses type to create the class. But if you define __metaclass__ to point to a callable, Python will call __metaclass__() after the initial creation of the class object, passing in the class object, the class name, the list of base classes and the namespace dictionary.
Python 2.x also allows you to assign to the global __metaclass__ hook, which will be used if there is not a class-local __metaclass__ hook (is there an equivalent in Python 3?).
Thus, the basic process of metaclass programming looks like this:
# Metaprogramming/SimpleMeta1.py
# Two-step metaclass creation in Python 2.x
class SimpleMeta1(type):
def __init__(cls, name, bases, nmspc):
super(SimpleMeta1, cls).__init__(name, bases, nmspc)
cls.uses_metaclass = lambda self : "Yes!"
class Simple1(object):
__metaclass__ = SimpleMeta1
def foo(self): pass
@staticmethod
def bar(): pass
simple = Simple1()
print([m for m in dir(simple) if not m.startswith('__')])
# A new method has been injected by the metaclass:
print simple.uses_metaclass()
""" Output:
['bar', 'foo', 'uses_metaclass']
Yes!
"""
By convention, when defining metaclasses cls is used rather than self as the first argument to all methods except __new__() (which uses mcl, for reasons explained later). cls is the class object that is being modified.
Note that the practice of calling the base-class constructor first (via super()) in the derived-class constructor should be followed with metaclasses as well.
__metaclass__ only needs to be callable, so in Python 2.x it's possible to define __metaclass__ inline:
# Metaprogramming/SimpleMeta2.py
# Combining the steps for metaclass creation in Python 2.x
class Simple2(object):
class __metaclass__(type):
def __init__(cls, name, bases, nmspc):
# This won't work:
# super(__metaclass__, cls).__init__(name, bases, nmspc)
# Less-flexible specific call:
type.__init__(cls, name, bases, nmspc)
cls.uses_metaclass = lambda self : "Yes!"
class Simple3(Simple2): pass
simple = Simple3()
print simple.uses_metaclass()
""" Output:
Yes!
"""
The compiler won't accept the super() call because it says __metaclass__ hasn't been defined, forcing us to use the specific call to type.__init__().
Because it only needs to be callable, it's even possible to define __metaclass__ as a function:
# Metaprogramming/SimpleMeta3.py
# A function for __metaclass__ in Python 2.x
class Simple4(object):
def __metaclass__(name, bases, nmspc):
cls = type(name, bases, nmspc)
cls.uses_metaclass = lambda self : "Yes!"
return cls
simple = Simple4()
print simple.uses_metaclass()
""" Output:
Yes!
"""
As you'll see, Python 3 doesn't allow the syntax of these last two examples. Even so, the above example makes it quite clear what's happening: the class object is created, then modified, then returned.
Note
Or does it allow that syntax?
The Metaclass Hook in Python 3
Python 3 changes the metaclass hook. It doesn't disallow the __metaclass__ field, but it ignores it. Instead, you use a keyword argument in the base-class list:
class Simple1(object, metaclass = SimpleMeta1):
...
This means that none of the (clever) alternative ways of defining __metaclass__ directly as a class or function are available in Python 3 [[check this]]. All metaclasses must be defined as separate classes. This is probably just as well, as it makes metaclass programs more consistent and thus easier to read and understand.
Example: Self-Registration of Subclasses
It is sometimes convienient to use inheritance as an organizing mechanism ? each sublclass becomes an element of a group that you work on. For example, in CodeManager.py in the Comprehensions chapter, the subclasses of Language were all the languages that needed to be processed. Each Language subclass described specific processing traits for that language.
To solve this problem, consider a system that automatically keeps a list of all of its "leaf" subclasses (only the classes that have no inheritors). This way we can easily enumerate through all the subtypes:
# Metaprogramming/RegisterLeafClasses.py
class RegisterLeafClasses(type):
def __init__(cls, name, bases, nmspc):
super(RegisterLeafClasses, cls).__init__(name, bases, nmspc)
if not hasattr(cls, 'registry'):
cls.registry = set()
cls.registry.add(cls)
cls.registry -= set(bases) # Remove base classes
# Metamethods, called on class objects:
def __iter__(cls):
return iter(cls.registry)
def __str__(cls):
if cls in cls.registry:
return cls.__name__
return cls.__name__ + ": " + ", ".join([sc.__name__ for sc in cls])
class Color(object):
__metaclass__ = RegisterLeafClasses
class Blue(Color): pass
class Red(Color): pass
class Green(Color): pass
class Yellow(Color): pass
print(Color)
class PhthaloBlue(Blue): pass
class CeruleanBlue(Blue): pass
print(Color)
for c in Color: # Iterate over subclasses
print(c)
class Shape(object):
__metaclass__ = RegisterLeafClasses
class Round(Shape): pass
class Square(Shape): pass
class Triangular(Shape): pass
class Boxy(Shape): pass
print(Shape)
class Circle(Round): pass
class Ellipse(Round): pass
print(Shape)
""" Output:
Color: Red, Blue, Green, Yellow
Color: Red, CeruleanBlue, Green, PhthaloBlue, Yellow
Red
CeruleanBlue
Green
PhthaloBlue
Yellow
Shape: Square, Round, Boxy, Triangular
Shape: Square, Ellipse, Circle, Boxy, Triangular
"""
Two separate tests are used to show that the registries are independent of each other. Each test shows what happens when another level of leaf classes are added ? the former leaf becomes a base class, and so is removed from the registry.
This also introduces metamethods, which are defined in the metaclass so that they become methods of the class. That is, you call them on the class rather than object instances, and their first argument is the class object rather than self.
Using Class Decorators
Using the inspect module
(As in the Comprehensions chapter)
Example: Making a Class "Final"
It is sometimes convenient to prevent a class from being inherited:
# Metaprogramming/Final.py
# Emulating Java's 'final'
class final(type):
def __init__(cls, name, bases, namespace):
super(final, cls).__init__(name, bases, namespace)
for klass in bases:
if isinstance(klass, final):
raise TypeError(str(klass.__name__) + " is final")
class A(object):
pass
class B(A):
__metaclass__= final
print B.__bases__
print isinstance(B, final)
# Produces compile-time error:
class C(B):
pass
""" Output:
(<class '__main__.A'>,)
True
...
TypeError: B is final
"""
During class object creation, we check to see if any of the bases are derived from final. Notice that using a metaclass makes the new type an instance of that metaclass, even though the metaclass doesn't show up in the base-class list.
Because this process of checking for finality must be installed to happen as the subclasses are created, rather than afterwards as performed by class decorators, it appears that this is an example of something that requires metaclasses and can't be accomplished with class decorators.
Using __init__ vs. __new__ in Metaclasses
It can be confusing when you see metaclass examples that appear to arbitrarily use __new__ or __init__ ? why choose one over the other?
__new__ is called for the creation of a new class, while __init__ is called after the class is created, to perform additional initialization before the class is handed to the caller:
# Metaprogramming/NewVSInit.py
from pprint import pprint
class Tag1: pass
class Tag2: pass
class Tag3:
def tag3_method(self): pass
class MetaBase(type):
def __new__(mcl, name, bases, nmspc):
print('MetaBase.__new__\n')
return super(MetaBase, mcl).__new__(mcl, name, bases, nmspc)
def __init__(cls, name, bases, nmspc):
print('MetaBase.__init__\n')
super(MetaBase, cls).__init__(name, bases, nmspc)
class MetaNewVSInit(MetaBase):
def __new__(mcl, name, bases, nmspc):
# First argument is the metaclass ``MetaNewVSInit``
print('MetaNewVSInit.__new__')
for x in (mcl, name, bases, nmspc): pprint(x)
print('')
# These all work because the class hasn't been created yet:
if 'foo' in nmspc: nmspc.pop('foo')
name += '_x'
bases += (Tag1,)
nmspc['baz'] = 42
return super(MetaNewVSInit, mcl).__new__(mcl, name, bases, nmspc)
def __init__(cls, name, bases, nmspc):
# First argument is the class being initialized
print('MetaNewVSInit.__init__')
for x in (cls, name, bases, nmspc): pprint(x)
print('')
if 'bar' in nmspc: nmspc.pop('bar') # No effect
name += '_y' # No effect
bases += (Tag2,) # No effect
nmspc['pi'] = 3.14159 # No effect
super(MetaNewVSInit, cls).__init__(name, bases, nmspc)
# These do work because they operate on the class object:
cls.__name__ += '_z'
cls.__bases__ += (Tag3,)
cls.e = 2.718
class Test(object):
__metaclass__ = MetaNewVSInit
def __init__(self):
print('Test.__init__')
def foo(self): print('foo still here')
def bar(self): print('bar still here')
t = Test()
print('class name: ' + Test.__name__)
print('base classes: ', [c.__name__ for c in Test.__bases__])
print([m for m in dir(t) if not m.startswith("__")])
t.bar()
print(t.e)
""" Output:
MetaNewVSInit.__new__
<class '__main__.MetaNewVSInit'>
'Test'
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
'__metaclass__': <class '__main__.MetaNewVSInit'>,
'__module__': '__main__',
'bar': <function bar at 0x7ed70>,
'foo': <function foo at 0x7ed30>}
MetaBase.__new__
MetaNewVSInit.__init__
<class '__main__.Test_x'>
'Test'
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
'__metaclass__': <class '__main__.MetaNewVSInit'>,
'__module__': '__main__',
'bar': <function bar at 0x7ed70>,
'baz': 42}
MetaBase.__init__
Test.__init__
class name: Test_x_z
('base classes: ', ['object', 'Tag1', 'Tag3'])
['bar', 'baz', 'e', 'tag3_method']
bar still here
2.718
"""
The primary difference is that when overriding __new__() you can change things like the 'name', 'bases' and 'namespace' arguments before you call the super constructor and it will have an effect, but doing the same thing in __init__() you won't get any results from the constructor call.
One special case in __new__() is that you can manipulate things like __slots__, but in __init__() you can't.
Note that, since the base-class version of __init__() doesn't make any modifications, it makes sense to call it first, then perform any additional operations. In C++ and Java, the base-class constructor must be called as the first operation in a derived-class constructor, which makes sense because derived-class constructions can then build upon base-class foundations.
In many cases, the choice of __new__() vs __init__() is a style issue and doesn't matter, but because __new__() can do everything and __init__() is slightly more limited, some people just start using __new__() and stick with it. This use can be confusing ? I tend to hunt for the reason that __init__() has been chosen, and if I can't find it wonder whether the author knew what they were doing. I prefer to only use __new__() when it has meaning ? when you must in order to change things that only __new__() can change.
Class Methods and Metamethods
A metamethod can be called from either the metaclass or from the class, but not from an instance. A classmethod can be called from either a class or its instances, but is not part of the metaclass.
(Is a similar relationship true with attributes, or is it different?)
Intercepting Class Creation
This example implements Singleton using metaclasses, by overriding the __call__() metamethod, which is invoked when a new instance is created:
# Metaprogramming/Singleton.py
class Singleton(type):
instance = None
def __call__(cls, *args, **kw):
if not cls.instance:
cls.instance = super(Singleton, cls).__call__(*args, **kw)
return cls.instance
class ASingleton(object):
__metaclass__ = Singleton
a = ASingleton()
b = ASingleton()
assert a is b
print(a.__class__.__name__, b.__class__.__name__)
class BSingleton(object):
__metaclass__ = Singleton
c = BSingleton()
d = BSingleton()
assert c is d
print(c.__class__.__name__, d.__class__.__name__)
assert c is not a
""" Output:
('ASingleton', 'ASingleton')
('BSingleton', 'BSingleton')
"""
By overriding __call__() in the metaclass, the creation of instances are intercepted. Instance creation is bypassed if one already exists.
Note the dependence upon the behavior of static class fields. When cls.instance is first read, it gets the static value of instance from the metaclass, which is None. However, when the assignment is made, Python creates a local version for the particular class, and the next time cls.instance is read, it sees that local version. Because of this behavior, each class ends up with its own class-specific instance field (thus instance is not somehow being "inherited" from the metaclass).
A Class Decorator Singleton
# Metaprogramming/SingletonDecorator.py
def singleton(klass):
"Simple replacement of object creation operation"
def getinstance(*args, **kw):
if not hasattr(klass, 'instance'):
klass.instance = klass(*args, **kw)
return klass.instance
return getinstance
def singleton(klass):
"""
More powerful approach: Change the behavior
of the instances AND the class object.
"""
class Decorated(klass):
def __init__(self, *args, **kwargs):
if hasattr(klass, '__init__'):
klass.__init__(self, *args, **kwargs)
def __repr__(self) : return klass.__name__ + " obj"
__str__ = __repr__
Decorated.__name__ = klass.__name__
class ClassObject:
def __init__(cls):
cls.instance = None
def __repr__(cls):
return klass.__name__
__str__ = __repr__
def __call__(cls, *args, **kwargs):
print str(cls) + " __call__ "
if not cls.instance:
cls.instance = Decorated(*args, **kwargs)
return cls.instance
return ClassObject()
@singleton
class ASingleton: pass
a = ASingleton()
b = ASingleton()
print(a, b)
print a.__class__.__name__
print ASingleton
assert a is b
@singleton
class BSingleton:
def __init__(self, x):
self.x = x
c = BSingleton(11)
d = BSingleton(22)
assert c is d
assert c is not a
""" Output:
ASingleton __call__
ASingleton __call__
(ASingleton obj, ASingleton obj)
ASingleton
ASingleton
BSingleton __call__
BSingleton __call__
"""
The __prepare__() Metamethod
One of the things you can't do with class decorators is to replace the default dictionary. In Python 3 this is enabled with the __prepare__() metamethod:
@classmethod
def __prepare__(mcl, name, bases):
return odict()
For an example of using both __prepare__() and __slots__ in metaclasses, see Michele Simionato's article.
Module-level __metaclass__ Assignment
(Does this work in Python 3? If not is there an alternative?)
Metaclass Conflicts
Note that the metaclass argument is singular ? you can't attach more than one metaclass to a class. However, through multiple inheritance you can accidentally end up with more than one metaclass, and this produces a conflict which must be resolved.
http://code.activestate.com/recipes/204197/
Further Reading
Excellent step-by-step introduction to metaclasses:
http://cleverdevil.org/computing/78/
Metaclass intro and comparison of syntax between Python 2.x and 3.x:
http://mikewatkins.ca/2008/11/29/python-2-and-3-metaclasses/
David Mertz's metaclass primer:
http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html
Three-part in-depth coverage of metaclasses on IBM Developer Works. Quite useful and authoritative:
http://www.ibm.com/developerworks/linux/library/l-pymeta.html
http://www.ibm.com/developerworks/linux/library/l-pymeta2/
http://www.ibm.com/developerworks/linux/library/l-pymeta3.html
Michele Simionato's articles on Artima, with special emphasis on the difference between Python 2.x and 3.x metaclasses:
http://www.artima.com/weblogs/viewpost.jsp?thread=236234
http://www.artima.com/weblogs/viewpost.jsp?thread=236260
Once you understand the foundations, you can find lots of examples by searching for "metaclass" within the Python Cookbook: http://code.activestate.com/recipes/langs/python/
The printed version of the Python Cookbook has far fewer examples than the online version, but the print version has been filtered and edited and so tends to be more authoritative.
Ian Bicking writes about metaclasses:
http://blog.ianbicking.org/a-conservative-metaclass.html
http://blog.ianbicking.org/metaclass-fun.html
http://blog.ianbicking.org/A-Declarative-Syntax-Extension.html
http://blog.ianbicking.org/self-take-two.html
Lots of good information about classes, types, metaclasses, etc., including historical stuff in the Python 2.2 docs (is this duplicated in later versions of the docs):
http://www.python.org/download/releases/2.2/descrintro/
Classes are just objects, and they can be modified the same way:
>>> class Foo: pass
...
>>> Foo.field = 42
>>> x = Foo()
>>> x.field
42
>>> Foo.field2 = 99
>>> x.field2
99
>>> Foo.method = lambda self: "Hi!"
>>> x.method()
'Hi!'
To modify a class, you perform operations on it like any other object. You can add and subtract fields and methods, for example. The difference is that any change you make to a class affects all the objects of that class, even the ones that have already been instantiated.
What creates these special "class" objects? Other special objects, called metaclasses.
The default metaclass is called type and in the vast majority of cases it does the right thing. In some situations, however, you can gain leverage by modifying the way that classes are produced ? typically by performing extra actions or injecting code. When this is the case, you can use metaclass programming to modify the way that some of your class objects are created.
It's worth re-emphasizing that in the vast majority of cases, you don't need metaclasses, because it's a fascinating toy and the temptation to use it everywhere can be overwhelming. Some of the examples in this chapter will show both metaclass and non-metaclass solutions to a problem, so you can see that there's usually another (often simpler) approach.
Some of the functionality that was previously only available with metaclasses is now available in a simpler form using class decorators. It is still useful, however, to understand metaclasses, and certain results can still be achieved only through metaclass programming.
Basic Metaprogramming
So metaclasses create classes, and classes create instances. Normally when we write a class, the default metaclass type is automatically invoked to create that class, and we aren't even aware that it's happening.
It's possible to explicitly code the metaclass' creation of a class. type called with one argument produces the type information of an existing class; type called with three arguments creates a new class object. The arguments when invoking type are the name of the class, a list of base classes, and a dictionary giving the namespace for the class (all the fields and methods). So the equivalent of:
class C: pass
is:
C = type('C', (), {})
Classes are often referred to as "types," so this reads fairly sensibly: you're calling a function that creates a new type based on its arguments.
We can also add base classes, fields and methods:
# Metaprogramming/MyList.py
def howdy(self, you):
print("Howdy, " + you)
MyList = type('MyList', (list,), dict(x=42, howdy=howdy))
ml = MyList()
ml.append("Camembert")
print(ml)
print(ml.x)
ml.howdy("John")
print(ml.__class__.__class__)
""" Output:
['Camembert']
42
Howdy, John
"""
Note that printing the class of the class produces the metaclass.
The ability to generate classes programmatically using type opens up some interesting possibilities. Consider the GreenHouseLanguage.py example in the Jython chapter ? all the subclasses in that case were written using repetetive code. We can automate the generation of the subclasses using type:
# Metaprogramming/GreenHouse.py
class Event(object):
events = [] # static
def __init__(self, action, time):
self.action = action
self.time = time
Event.events.append(self)
def __cmp__ (self, other):
"So sort() will compare only on time."
return cmp(self.time, other.time)
def run(self):
print("%.2f: %s" % (self.time, self.action))
@staticmethod
def run_events():
Event.events.sort();
for e in Event.events:
e.run()
def create_mc(description):
"Create subclass using the 'type' metaclass"
class_name = "".join(x.capitalize() for x in description.split())
def __init__(self, time):
Event.__init__(self, description + " [mc]", time)
globals()[class_name] = \
type(class_name, (Event,), dict(__init__ = __init__))
def create_exec(description):
"Create subclass by exec-ing a string"
class_name = "".join(x.capitalize() for x in description.split())
klass = """
class %s(Event):
def __init__(self, time):
Event.__init__(self, "%s [exec]", time)
""" % (class_name, description)
exec klass in globals()
if __name__ == "__main__":
descriptions = ["Light on", "Light off", "Water on", "Water off",
"Thermostat night", "Thermostat day", "Ring bell"]
initializations = "ThermostatNight(5.00); LightOff(2.00); \
WaterOn(3.30); WaterOff(4.45); LightOn(1.00); \
RingBell(7.00); ThermostatDay(6.00)"
[create_mc(dsc) for dsc in descriptions]
exec initializations in globals()
[create_exec(dsc) for dsc in descriptions]
exec initializations in globals()
Event.run_events()
""" Output:
1.00: Light on [mc]
1.00: Light on [exec]
2.00: Light off [mc]
2.00: Light off [exec]
3.30: Water on [mc]
3.30: Water on [exec]
4.45: Water off [mc]
4.45: Water off [exec]
5.00: Thermostat night [mc]
5.00: Thermostat night [exec]
6.00: Thermostat day [mc]
6.00: Thermostat day [exec]
7.00: Ring bell [mc]
7.00: Ring bell [exec]
"""
The Event base class is the same. The classes are created automatically using the create_mc() function, which takes its description argument and generates a class name from it. Then it defines an __init__() method, which it puts into the namespace dictionary for the type call, producing a new subclass of Event. Note that the resulting class must be inserted into the global namespace, otherwise it will not be seen.
This approach works fine, but then consider the subsequent create_exec() function, which accomplishes the same thing by calling exec on a string defining the class. This will be much easier to understand by the vast majority of the people reading your code: those who do not understand metaclasses.
The Metaclass Hook
So far, we've only used the type metaclass directly. Metaclass programming involves hooking our own operations into the creation of class objects. This is accomplished by:
Writing a subclass of the metaclass type.
Inserting the new metaclass into the class creation process using the metaclass hook.
In Python 2.x, the metaclass hook is a static field in the class called __metaclass__. In the ordinary case, this is not assigned so Python just uses type to create the class. But if you define __metaclass__ to point to a callable, Python will call __metaclass__() after the initial creation of the class object, passing in the class object, the class name, the list of base classes and the namespace dictionary.
Python 2.x also allows you to assign to the global __metaclass__ hook, which will be used if there is not a class-local __metaclass__ hook (is there an equivalent in Python 3?).
Thus, the basic process of metaclass programming looks like this:
# Metaprogramming/SimpleMeta1.py
# Two-step metaclass creation in Python 2.x
class SimpleMeta1(type):
def __init__(cls, name, bases, nmspc):
super(SimpleMeta1, cls).__init__(name, bases, nmspc)
cls.uses_metaclass = lambda self : "Yes!"
class Simple1(object):
__metaclass__ = SimpleMeta1
def foo(self): pass
@staticmethod
def bar(): pass
simple = Simple1()
print([m for m in dir(simple) if not m.startswith('__')])
# A new method has been injected by the metaclass:
print simple.uses_metaclass()
""" Output:
['bar', 'foo', 'uses_metaclass']
Yes!
"""
By convention, when defining metaclasses cls is used rather than self as the first argument to all methods except __new__() (which uses mcl, for reasons explained later). cls is the class object that is being modified.
Note that the practice of calling the base-class constructor first (via super()) in the derived-class constructor should be followed with metaclasses as well.
__metaclass__ only needs to be callable, so in Python 2.x it's possible to define __metaclass__ inline:
# Metaprogramming/SimpleMeta2.py
# Combining the steps for metaclass creation in Python 2.x
class Simple2(object):
class __metaclass__(type):
def __init__(cls, name, bases, nmspc):
# This won't work:
# super(__metaclass__, cls).__init__(name, bases, nmspc)
# Less-flexible specific call:
type.__init__(cls, name, bases, nmspc)
cls.uses_metaclass = lambda self : "Yes!"
class Simple3(Simple2): pass
simple = Simple3()
print simple.uses_metaclass()
""" Output:
Yes!
"""
The compiler won't accept the super() call because it says __metaclass__ hasn't been defined, forcing us to use the specific call to type.__init__().
Because it only needs to be callable, it's even possible to define __metaclass__ as a function:
# Metaprogramming/SimpleMeta3.py
# A function for __metaclass__ in Python 2.x
class Simple4(object):
def __metaclass__(name, bases, nmspc):
cls = type(name, bases, nmspc)
cls.uses_metaclass = lambda self : "Yes!"
return cls
simple = Simple4()
print simple.uses_metaclass()
""" Output:
Yes!
"""
As you'll see, Python 3 doesn't allow the syntax of these last two examples. Even so, the above example makes it quite clear what's happening: the class object is created, then modified, then returned.
Note
Or does it allow that syntax?
The Metaclass Hook in Python 3
Python 3 changes the metaclass hook. It doesn't disallow the __metaclass__ field, but it ignores it. Instead, you use a keyword argument in the base-class list:
class Simple1(object, metaclass = SimpleMeta1):
...
This means that none of the (clever) alternative ways of defining __metaclass__ directly as a class or function are available in Python 3 [[check this]]. All metaclasses must be defined as separate classes. This is probably just as well, as it makes metaclass programs more consistent and thus easier to read and understand.
Example: Self-Registration of Subclasses
It is sometimes convienient to use inheritance as an organizing mechanism ? each sublclass becomes an element of a group that you work on. For example, in CodeManager.py in the Comprehensions chapter, the subclasses of Language were all the languages that needed to be processed. Each Language subclass described specific processing traits for that language.
To solve this problem, consider a system that automatically keeps a list of all of its "leaf" subclasses (only the classes that have no inheritors). This way we can easily enumerate through all the subtypes:
# Metaprogramming/RegisterLeafClasses.py
class RegisterLeafClasses(type):
def __init__(cls, name, bases, nmspc):
super(RegisterLeafClasses, cls).__init__(name, bases, nmspc)
if not hasattr(cls, 'registry'):
cls.registry = set()
cls.registry.add(cls)
cls.registry -= set(bases) # Remove base classes
# Metamethods, called on class objects:
def __iter__(cls):
return iter(cls.registry)
def __str__(cls):
if cls in cls.registry:
return cls.__name__
return cls.__name__ + ": " + ", ".join([sc.__name__ for sc in cls])
class Color(object):
__metaclass__ = RegisterLeafClasses
class Blue(Color): pass
class Red(Color): pass
class Green(Color): pass
class Yellow(Color): pass
print(Color)
class PhthaloBlue(Blue): pass
class CeruleanBlue(Blue): pass
print(Color)
for c in Color: # Iterate over subclasses
print(c)
class Shape(object):
__metaclass__ = RegisterLeafClasses
class Round(Shape): pass
class Square(Shape): pass
class Triangular(Shape): pass
class Boxy(Shape): pass
print(Shape)
class Circle(Round): pass
class Ellipse(Round): pass
print(Shape)
""" Output:
Color: Red, Blue, Green, Yellow
Color: Red, CeruleanBlue, Green, PhthaloBlue, Yellow
Red
CeruleanBlue
Green
PhthaloBlue
Yellow
Shape: Square, Round, Boxy, Triangular
Shape: Square, Ellipse, Circle, Boxy, Triangular
"""
Two separate tests are used to show that the registries are independent of each other. Each test shows what happens when another level of leaf classes are added ? the former leaf becomes a base class, and so is removed from the registry.
This also introduces metamethods, which are defined in the metaclass so that they become methods of the class. That is, you call them on the class rather than object instances, and their first argument is the class object rather than self.
Using Class Decorators
Using the inspect module
(As in the Comprehensions chapter)
Example: Making a Class "Final"
It is sometimes convenient to prevent a class from being inherited:
# Metaprogramming/Final.py
# Emulating Java's 'final'
class final(type):
def __init__(cls, name, bases, namespace):
super(final, cls).__init__(name, bases, namespace)
for klass in bases:
if isinstance(klass, final):
raise TypeError(str(klass.__name__) + " is final")
class A(object):
pass
class B(A):
__metaclass__= final
print B.__bases__
print isinstance(B, final)
# Produces compile-time error:
class C(B):
pass
""" Output:
(<class '__main__.A'>,)
True
...
TypeError: B is final
"""
During class object creation, we check to see if any of the bases are derived from final. Notice that using a metaclass makes the new type an instance of that metaclass, even though the metaclass doesn't show up in the base-class list.
Because this process of checking for finality must be installed to happen as the subclasses are created, rather than afterwards as performed by class decorators, it appears that this is an example of something that requires metaclasses and can't be accomplished with class decorators.
Using __init__ vs. __new__ in Metaclasses
It can be confusing when you see metaclass examples that appear to arbitrarily use __new__ or __init__ ? why choose one over the other?
__new__ is called for the creation of a new class, while __init__ is called after the class is created, to perform additional initialization before the class is handed to the caller:
# Metaprogramming/NewVSInit.py
from pprint import pprint
class Tag1: pass
class Tag2: pass
class Tag3:
def tag3_method(self): pass
class MetaBase(type):
def __new__(mcl, name, bases, nmspc):
print('MetaBase.__new__\n')
return super(MetaBase, mcl).__new__(mcl, name, bases, nmspc)
def __init__(cls, name, bases, nmspc):
print('MetaBase.__init__\n')
super(MetaBase, cls).__init__(name, bases, nmspc)
class MetaNewVSInit(MetaBase):
def __new__(mcl, name, bases, nmspc):
# First argument is the metaclass ``MetaNewVSInit``
print('MetaNewVSInit.__new__')
for x in (mcl, name, bases, nmspc): pprint(x)
print('')
# These all work because the class hasn't been created yet:
if 'foo' in nmspc: nmspc.pop('foo')
name += '_x'
bases += (Tag1,)
nmspc['baz'] = 42
return super(MetaNewVSInit, mcl).__new__(mcl, name, bases, nmspc)
def __init__(cls, name, bases, nmspc):
# First argument is the class being initialized
print('MetaNewVSInit.__init__')
for x in (cls, name, bases, nmspc): pprint(x)
print('')
if 'bar' in nmspc: nmspc.pop('bar') # No effect
name += '_y' # No effect
bases += (Tag2,) # No effect
nmspc['pi'] = 3.14159 # No effect
super(MetaNewVSInit, cls).__init__(name, bases, nmspc)
# These do work because they operate on the class object:
cls.__name__ += '_z'
cls.__bases__ += (Tag3,)
cls.e = 2.718
class Test(object):
__metaclass__ = MetaNewVSInit
def __init__(self):
print('Test.__init__')
def foo(self): print('foo still here')
def bar(self): print('bar still here')
t = Test()
print('class name: ' + Test.__name__)
print('base classes: ', [c.__name__ for c in Test.__bases__])
print([m for m in dir(t) if not m.startswith("__")])
t.bar()
print(t.e)
""" Output:
MetaNewVSInit.__new__
<class '__main__.MetaNewVSInit'>
'Test'
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
'__metaclass__': <class '__main__.MetaNewVSInit'>,
'__module__': '__main__',
'bar': <function bar at 0x7ed70>,
'foo': <function foo at 0x7ed30>}
MetaBase.__new__
MetaNewVSInit.__init__
<class '__main__.Test_x'>
'Test'
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
'__metaclass__': <class '__main__.MetaNewVSInit'>,
'__module__': '__main__',
'bar': <function bar at 0x7ed70>,
'baz': 42}
MetaBase.__init__
Test.__init__
class name: Test_x_z
('base classes: ', ['object', 'Tag1', 'Tag3'])
['bar', 'baz', 'e', 'tag3_method']
bar still here
2.718
"""
The primary difference is that when overriding __new__() you can change things like the 'name', 'bases' and 'namespace' arguments before you call the super constructor and it will have an effect, but doing the same thing in __init__() you won't get any results from the constructor call.
One special case in __new__() is that you can manipulate things like __slots__, but in __init__() you can't.
Note that, since the base-class version of __init__() doesn't make any modifications, it makes sense to call it first, then perform any additional operations. In C++ and Java, the base-class constructor must be called as the first operation in a derived-class constructor, which makes sense because derived-class constructions can then build upon base-class foundations.
In many cases, the choice of __new__() vs __init__() is a style issue and doesn't matter, but because __new__() can do everything and __init__() is slightly more limited, some people just start using __new__() and stick with it. This use can be confusing ? I tend to hunt for the reason that __init__() has been chosen, and if I can't find it wonder whether the author knew what they were doing. I prefer to only use __new__() when it has meaning ? when you must in order to change things that only __new__() can change.
Class Methods and Metamethods
A metamethod can be called from either the metaclass or from the class, but not from an instance. A classmethod can be called from either a class or its instances, but is not part of the metaclass.
(Is a similar relationship true with attributes, or is it different?)
Intercepting Class Creation
This example implements Singleton using metaclasses, by overriding the __call__() metamethod, which is invoked when a new instance is created:
# Metaprogramming/Singleton.py
class Singleton(type):
instance = None
def __call__(cls, *args, **kw):
if not cls.instance:
cls.instance = super(Singleton, cls).__call__(*args, **kw)
return cls.instance
class ASingleton(object):
__metaclass__ = Singleton
a = ASingleton()
b = ASingleton()
assert a is b
print(a.__class__.__name__, b.__class__.__name__)
class BSingleton(object):
__metaclass__ = Singleton
c = BSingleton()
d = BSingleton()
assert c is d
print(c.__class__.__name__, d.__class__.__name__)
assert c is not a
""" Output:
('ASingleton', 'ASingleton')
('BSingleton', 'BSingleton')
"""
By overriding __call__() in the metaclass, the creation of instances are intercepted. Instance creation is bypassed if one already exists.
Note the dependence upon the behavior of static class fields. When cls.instance is first read, it gets the static value of instance from the metaclass, which is None. However, when the assignment is made, Python creates a local version for the particular class, and the next time cls.instance is read, it sees that local version. Because of this behavior, each class ends up with its own class-specific instance field (thus instance is not somehow being "inherited" from the metaclass).
A Class Decorator Singleton
# Metaprogramming/SingletonDecorator.py
def singleton(klass):
"Simple replacement of object creation operation"
def getinstance(*args, **kw):
if not hasattr(klass, 'instance'):
klass.instance = klass(*args, **kw)
return klass.instance
return getinstance
def singleton(klass):
"""
More powerful approach: Change the behavior
of the instances AND the class object.
"""
class Decorated(klass):
def __init__(self, *args, **kwargs):
if hasattr(klass, '__init__'):
klass.__init__(self, *args, **kwargs)
def __repr__(self) : return klass.__name__ + " obj"
__str__ = __repr__
Decorated.__name__ = klass.__name__
class ClassObject:
def __init__(cls):
cls.instance = None
def __repr__(cls):
return klass.__name__
__str__ = __repr__
def __call__(cls, *args, **kwargs):
print str(cls) + " __call__ "
if not cls.instance:
cls.instance = Decorated(*args, **kwargs)
return cls.instance
return ClassObject()
@singleton
class ASingleton: pass
a = ASingleton()
b = ASingleton()
print(a, b)
print a.__class__.__name__
print ASingleton
assert a is b
@singleton
class BSingleton:
def __init__(self, x):
self.x = x
c = BSingleton(11)
d = BSingleton(22)
assert c is d
assert c is not a
""" Output:
ASingleton __call__
ASingleton __call__
(ASingleton obj, ASingleton obj)
ASingleton
ASingleton
BSingleton __call__
BSingleton __call__
"""
The __prepare__() Metamethod
One of the things you can't do with class decorators is to replace the default dictionary. In Python 3 this is enabled with the __prepare__() metamethod:
@classmethod
def __prepare__(mcl, name, bases):
return odict()
For an example of using both __prepare__() and __slots__ in metaclasses, see Michele Simionato's article.
Module-level __metaclass__ Assignment
(Does this work in Python 3? If not is there an alternative?)
Metaclass Conflicts
Note that the metaclass argument is singular ? you can't attach more than one metaclass to a class. However, through multiple inheritance you can accidentally end up with more than one metaclass, and this produces a conflict which must be resolved.
http://code.activestate.com/recipes/204197/
Further Reading
Excellent step-by-step introduction to metaclasses:
http://cleverdevil.org/computing/78/
Metaclass intro and comparison of syntax between Python 2.x and 3.x:
http://mikewatkins.ca/2008/11/29/python-2-and-3-metaclasses/
David Mertz's metaclass primer:
http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html
Three-part in-depth coverage of metaclasses on IBM Developer Works. Quite useful and authoritative:
http://www.ibm.com/developerworks/linux/library/l-pymeta.html
http://www.ibm.com/developerworks/linux/library/l-pymeta2/
http://www.ibm.com/developerworks/linux/library/l-pymeta3.html
Michele Simionato's articles on Artima, with special emphasis on the difference between Python 2.x and 3.x metaclasses:
http://www.artima.com/weblogs/viewpost.jsp?thread=236234
http://www.artima.com/weblogs/viewpost.jsp?thread=236260
Once you understand the foundations, you can find lots of examples by searching for "metaclass" within the Python Cookbook: http://code.activestate.com/recipes/langs/python/
The printed version of the Python Cookbook has far fewer examples than the online version, but the print version has been filtered and edited and so tends to be more authoritative.
Ian Bicking writes about metaclasses:
http://blog.ianbicking.org/a-conservative-metaclass.html
http://blog.ianbicking.org/metaclass-fun.html
http://blog.ianbicking.org/A-Declarative-Syntax-Extension.html
http://blog.ianbicking.org/self-take-two.html
Lots of good information about classes, types, metaclasses, etc., including historical stuff in the Python 2.2 docs (is this duplicated in later versions of the docs):
http://www.python.org/download/releases/2.2/descrintro/
How Metaclasses work technically in Python 2 and 3
A metaclass is a class/object which defines a type/class of other classes. In Python a metaclass can be a class, function or any object that supports calling an interface. This is because to create a class object; its metaclass is called with the class name, base classes and attributes (methods). When no metaclass is defined (which is usually the case), the default metaclass type is used.
For example:
Python 3.x
# Here __metaclass__ points to the metaclass object.
class ExampleClass(metaclass=type):
pass
Python 2.x
# Here __metaclass__ points to the metaclass object.
class ExampleClass(object):
__metaclass__ = type
pass
When a class is created, the interpreter:
Gets the name of the class.
Gets the base classes of the class.
Gets the metaclass of the class. If it is defined, it will use this first. Otherwise, it will check in the base classes for the metaclass. It it can't find a metaclass in the base class, the type object is used instead.
Gets the variables/attributes in the class and stores them as a dictionary.
Passes this information to metaclass as metaclass(name_of_class, base_classes, attributes_dictionary) and it returns a class object.
For example:
# type(name, base, attrs)
# name is the name of the class
# base is a tuple of base classes (all methods/attributes are inherited
# from these) attrs is a dictionary filled with the class attributes
classObject = type('ExampleClass', (object,) ,{})
When type is called, its __call__ method is called. This method in turn calls the __new__ and __init__ methods. The __new__ method creates a new object, whereas the __init__ method initializes it. We can easily play with methods. This is a working example:
Python 3.x
class a:
def __init__(self, data):
self.data = data
def getd3(self):
return self.data * 3
class MyMeta(type):
def __new__(metaname, classname, baseclasses, attrs):
print('New called with')
print('metaname', metaname)
print('classname', classname)
print('baseclasses', baseclasses)
print('attrs', attrs)
attrs['getdata'] = a.__dict__['getd3']
# attrs['getdata'] = a.getd3
return type.__new__(metaname, classname, baseclasses, attrs)
def __init__(classobject, classname, baseclasses, attrs):
print('init called with')
print('classobject', classobject)
print('classname', classname)
print('baseclasses', baseclasses)
print('attrs', attrs)
class Kls(metaclass=MyMeta):
def __init__(self,data):
self.data = data
def printd(self):
print(self.data)
ik = Kls('arun')
ik.printd()
print(ik.getdata())
When running the code, we get:
New called with
metaname <class '__main__.MyMeta'>
classname Kls
baseclasses ()
attrs {'__module__': '__main__', 'printd': <function printd at 0x7f3ebca86958>, '__init__': <function __init__ at 0x7f3ebca868d0>}
init called with
classobject <class '__main__.Kls'>
classname Kls
baseclasses ()
attrs {'__module__': '__main__', 'getdata': <function getd3 at 0x7f3ebca86408>, 'printd': <function printd at 0x7f3ebca86958>, '__init__': <function __init__ at 0x7f3ebca868d0>}
arun
arunarunarun
Python 2.x
class a(object):
def __init__(self, data):
self.data = data
def getd3(self):
return self.data * 3
class MyMeta(type):
def __new__(metaname, classname, baseclasses, attrs):
print 'New called with'
print 'metaname', metaname
print 'classname', classname
print 'baseclasses', baseclasses
print 'attrs', attrs
attrs['getdata'] = a.__dict__['getd3']
# attrs['getdata'] = a.getd3
return type.__new__(metaname, classname, baseclasses, attrs)
def __init__(classobject, classname, baseclasses, attrs):
print 'init called with'
print 'classobject', classobject
print 'classname', classname
print 'baseclasses', baseclasses
print 'attrs', attrs
class Kls(object):
__metaclass__ = MyMeta
def __init__(self, data):
self.data = data
def printd(self):
print self.data
ik = Kls('arun')
ik.printd()
print ik.getdata()
When running the code, we get:
New called with
metaname <class '__main__.MyMeta'>
classname Kls
baseclasses (<type 'object'>,)
attrs {'__module__': '__main__', '__metaclass__': <class '__main__.MyMeta'>, 'printd': <function printd at 0x7fbdab0176e0>, '__init__': <function __init__ at 0x7fbdab017668>}
init called with
classobject <class '__main__.Kls'>
classname Kls
baseclasses (<type 'object'>,)
attrs {'__module__': '__main__', 'getdata': <function getd3 at 0x7fbdab017500>, '__metaclass__': <class '__main__.MyMeta'>, 'printd': <function printd at 0x7fbdab0176e0>, '__init__': <function __init__ at 0x7fbdab017668>}
arun
arunarunarun
Normally we need to override only one method __new__ or __init__. We can also use function instead of a class. Here is an example:
Python 3.x
def meta_func(name, bases, attrs):
print('meta function called with', name, bases, attrs)
nattrs = {'mod' + key:attrs[key] for key in attrs}
return type(name, bases, nattrs)
MyMeta = meta_func
class Kls(metaclass=MyMeta):
def setd(self, data):
self.data = data
def getd(self):
return self.data
k = Kls()
k.modsetd('arun')
print(k.modgetd())
Gives us the following output:
meta function called with Kls () {'setd': <function setd at 0x7f3bafe7cd10>, '__module__': '__main__', 'getd': <function getd at 0x7f3bafe7cd98>}
arun
Python 2.x
def meta_func(name, bases, attrs):
print 'meta function called with', name, bases, attrs
nattrs = {'mod' + key:attrs[key] for key in attrs}
return type(name, bases, nattrs)
MyMeta = meta_func
class Kls(object):
__metaclass__ = MyMeta
def setd(self, data):
self.data = data
def getd(self):
return self.data
k = Kls()
k.modsetd('arun')
print k.modgetd()
Gives us the following output:
meta function called with Kls (<type 'object'>,) {'setd': <function setd at 0x88b21ec>, 'getd': <function getd at 0x88b22cc>, '__module__': '__main__', '__metaclass__': <function meta_func at 0xb72341b4>}
arun
Other then modifying base classes and methods of classes to be created, metaclasses can also modify instance creation process. This is because when we create an instance (ik = Kls()), this is like calling the class Kls. One point to note is that whenever we call an object its type's __call__ method is called. So in this case the class type is metaclass hence its __call__ method will be called. We can check like this:
Python 3.x
class MyMeta(type):
def __call__(clsname, *args):
print('MyMeta called with')
print('clsname:', clsname)
print('args:', args)
instance = object.__new__(clsname)
instance.__init__(*args)
return instance
class Kls(metaclass=MyMeta):
def __init__(self, data):
self.data = data
def printd(self):
print(self.data)
ik = Kls('arun')
ik.printd()
Python 2.x
class MyMeta(type):
def __call__(clsname, *args):
print 'MyMeta called with'
print 'clsname:', clsname
print 'args:' ,args
instance = object.__new__(clsname)
instance.__init__(*args)
return instance
class Kls(object):
__metaclass__ = MyMeta
def __init__(self,data):
self.data = data
def printd(self):
print self.data
ik = Kls('arun')
ik.printd()
The output is as follows:
MyMeta called with
clsname: <class '__main__.Kls'>
args: ('arun',)
arun
Equipped with this information, if we go to the start of our discussion about the class creation process, it ended with a call to the metaclass object, which provided a class object. It was like this:
Kls = MetaClass(name, bases, attrs)
Hence this call should call the metaclass's type. The metaclass type is the metaclass's metaclass! We can check this as follows:
Python 3.x
class SuperMeta(type):
def __call__(metaname, clsname, baseclasses, attrs):
print('SuperMeta Called')
clsob = type.__new__(metaname, clsname, baseclasses, attrs)
type.__init__(clsob, clsname, baseclasses, attrs)
return clsob
class MyMeta(type, metaclass=SuperMeta):
def __call__(cls, *args, **kwargs):
print('MyMeta called', cls, args, kwargs)
ob = object.__new__(cls, *args)
ob.__init__(*args)
return ob
print('create class')
class Kls(metaclass=MyMeta):
def __init__(self, data):
self.data = data
def printd(self):
print(self.data)
print('class created')
ik = Kls('arun')
ik.printd()
ik2 = Kls('avni')
ik2.printd()
Python 2.x
class SuperMeta(type):
def __call__(metaname, clsname, baseclasses, attrs):
print 'SuperMeta Called'
clsob = type.__new__(metaname, clsname, baseclasses, attrs)
type.__init__(clsob, clsname, baseclasses, attrs)
return clsob
class MyMeta(type):
__metaclass__ = SuperMeta
def __call__(cls, *args, **kwargs):
print 'MyMeta called', cls, args, kwargs
ob = object.__new__(cls, *args)
ob.__init__(*args)
return ob
print 'create class'
class Kls(object):
__metaclass__ = MyMeta
def __init__(self, data):
self.data = data
def printd(self):
print self.data
print 'class created'
ik = Kls('arun')
ik.printd()
ik2 = Kls('avni')
ik2.printd()
Gives us the following output:
create class
SuperMeta Called
class created
MyMeta called class '__main__.Kls' ('arun',) {}
arun
MyMeta called <class '__main__.Kls' ('avni',) {}
avni
For example:
Python 3.x
# Here __metaclass__ points to the metaclass object.
class ExampleClass(metaclass=type):
pass
Python 2.x
# Here __metaclass__ points to the metaclass object.
class ExampleClass(object):
__metaclass__ = type
pass
When a class is created, the interpreter:
Gets the name of the class.
Gets the base classes of the class.
Gets the metaclass of the class. If it is defined, it will use this first. Otherwise, it will check in the base classes for the metaclass. It it can't find a metaclass in the base class, the type object is used instead.
Gets the variables/attributes in the class and stores them as a dictionary.
Passes this information to metaclass as metaclass(name_of_class, base_classes, attributes_dictionary) and it returns a class object.
For example:
# type(name, base, attrs)
# name is the name of the class
# base is a tuple of base classes (all methods/attributes are inherited
# from these) attrs is a dictionary filled with the class attributes
classObject = type('ExampleClass', (object,) ,{})
When type is called, its __call__ method is called. This method in turn calls the __new__ and __init__ methods. The __new__ method creates a new object, whereas the __init__ method initializes it. We can easily play with methods. This is a working example:
Python 3.x
class a:
def __init__(self, data):
self.data = data
def getd3(self):
return self.data * 3
class MyMeta(type):
def __new__(metaname, classname, baseclasses, attrs):
print('New called with')
print('metaname', metaname)
print('classname', classname)
print('baseclasses', baseclasses)
print('attrs', attrs)
attrs['getdata'] = a.__dict__['getd3']
# attrs['getdata'] = a.getd3
return type.__new__(metaname, classname, baseclasses, attrs)
def __init__(classobject, classname, baseclasses, attrs):
print('init called with')
print('classobject', classobject)
print('classname', classname)
print('baseclasses', baseclasses)
print('attrs', attrs)
class Kls(metaclass=MyMeta):
def __init__(self,data):
self.data = data
def printd(self):
print(self.data)
ik = Kls('arun')
ik.printd()
print(ik.getdata())
When running the code, we get:
New called with
metaname <class '__main__.MyMeta'>
classname Kls
baseclasses ()
attrs {'__module__': '__main__', 'printd': <function printd at 0x7f3ebca86958>, '__init__': <function __init__ at 0x7f3ebca868d0>}
init called with
classobject <class '__main__.Kls'>
classname Kls
baseclasses ()
attrs {'__module__': '__main__', 'getdata': <function getd3 at 0x7f3ebca86408>, 'printd': <function printd at 0x7f3ebca86958>, '__init__': <function __init__ at 0x7f3ebca868d0>}
arun
arunarunarun
Python 2.x
class a(object):
def __init__(self, data):
self.data = data
def getd3(self):
return self.data * 3
class MyMeta(type):
def __new__(metaname, classname, baseclasses, attrs):
print 'New called with'
print 'metaname', metaname
print 'classname', classname
print 'baseclasses', baseclasses
print 'attrs', attrs
attrs['getdata'] = a.__dict__['getd3']
# attrs['getdata'] = a.getd3
return type.__new__(metaname, classname, baseclasses, attrs)
def __init__(classobject, classname, baseclasses, attrs):
print 'init called with'
print 'classobject', classobject
print 'classname', classname
print 'baseclasses', baseclasses
print 'attrs', attrs
class Kls(object):
__metaclass__ = MyMeta
def __init__(self, data):
self.data = data
def printd(self):
print self.data
ik = Kls('arun')
ik.printd()
print ik.getdata()
When running the code, we get:
New called with
metaname <class '__main__.MyMeta'>
classname Kls
baseclasses (<type 'object'>,)
attrs {'__module__': '__main__', '__metaclass__': <class '__main__.MyMeta'>, 'printd': <function printd at 0x7fbdab0176e0>, '__init__': <function __init__ at 0x7fbdab017668>}
init called with
classobject <class '__main__.Kls'>
classname Kls
baseclasses (<type 'object'>,)
attrs {'__module__': '__main__', 'getdata': <function getd3 at 0x7fbdab017500>, '__metaclass__': <class '__main__.MyMeta'>, 'printd': <function printd at 0x7fbdab0176e0>, '__init__': <function __init__ at 0x7fbdab017668>}
arun
arunarunarun
Normally we need to override only one method __new__ or __init__. We can also use function instead of a class. Here is an example:
Python 3.x
def meta_func(name, bases, attrs):
print('meta function called with', name, bases, attrs)
nattrs = {'mod' + key:attrs[key] for key in attrs}
return type(name, bases, nattrs)
MyMeta = meta_func
class Kls(metaclass=MyMeta):
def setd(self, data):
self.data = data
def getd(self):
return self.data
k = Kls()
k.modsetd('arun')
print(k.modgetd())
Gives us the following output:
meta function called with Kls () {'setd': <function setd at 0x7f3bafe7cd10>, '__module__': '__main__', 'getd': <function getd at 0x7f3bafe7cd98>}
arun
Python 2.x
def meta_func(name, bases, attrs):
print 'meta function called with', name, bases, attrs
nattrs = {'mod' + key:attrs[key] for key in attrs}
return type(name, bases, nattrs)
MyMeta = meta_func
class Kls(object):
__metaclass__ = MyMeta
def setd(self, data):
self.data = data
def getd(self):
return self.data
k = Kls()
k.modsetd('arun')
print k.modgetd()
Gives us the following output:
meta function called with Kls (<type 'object'>,) {'setd': <function setd at 0x88b21ec>, 'getd': <function getd at 0x88b22cc>, '__module__': '__main__', '__metaclass__': <function meta_func at 0xb72341b4>}
arun
Other then modifying base classes and methods of classes to be created, metaclasses can also modify instance creation process. This is because when we create an instance (ik = Kls()), this is like calling the class Kls. One point to note is that whenever we call an object its type's __call__ method is called. So in this case the class type is metaclass hence its __call__ method will be called. We can check like this:
Python 3.x
class MyMeta(type):
def __call__(clsname, *args):
print('MyMeta called with')
print('clsname:', clsname)
print('args:', args)
instance = object.__new__(clsname)
instance.__init__(*args)
return instance
class Kls(metaclass=MyMeta):
def __init__(self, data):
self.data = data
def printd(self):
print(self.data)
ik = Kls('arun')
ik.printd()
Python 2.x
class MyMeta(type):
def __call__(clsname, *args):
print 'MyMeta called with'
print 'clsname:', clsname
print 'args:' ,args
instance = object.__new__(clsname)
instance.__init__(*args)
return instance
class Kls(object):
__metaclass__ = MyMeta
def __init__(self,data):
self.data = data
def printd(self):
print self.data
ik = Kls('arun')
ik.printd()
The output is as follows:
MyMeta called with
clsname: <class '__main__.Kls'>
args: ('arun',)
arun
Equipped with this information, if we go to the start of our discussion about the class creation process, it ended with a call to the metaclass object, which provided a class object. It was like this:
Kls = MetaClass(name, bases, attrs)
Hence this call should call the metaclass's type. The metaclass type is the metaclass's metaclass! We can check this as follows:
Python 3.x
class SuperMeta(type):
def __call__(metaname, clsname, baseclasses, attrs):
print('SuperMeta Called')
clsob = type.__new__(metaname, clsname, baseclasses, attrs)
type.__init__(clsob, clsname, baseclasses, attrs)
return clsob
class MyMeta(type, metaclass=SuperMeta):
def __call__(cls, *args, **kwargs):
print('MyMeta called', cls, args, kwargs)
ob = object.__new__(cls, *args)
ob.__init__(*args)
return ob
print('create class')
class Kls(metaclass=MyMeta):
def __init__(self, data):
self.data = data
def printd(self):
print(self.data)
print('class created')
ik = Kls('arun')
ik.printd()
ik2 = Kls('avni')
ik2.printd()
Python 2.x
class SuperMeta(type):
def __call__(metaname, clsname, baseclasses, attrs):
print 'SuperMeta Called'
clsob = type.__new__(metaname, clsname, baseclasses, attrs)
type.__init__(clsob, clsname, baseclasses, attrs)
return clsob
class MyMeta(type):
__metaclass__ = SuperMeta
def __call__(cls, *args, **kwargs):
print 'MyMeta called', cls, args, kwargs
ob = object.__new__(cls, *args)
ob.__init__(*args)
return ob
print 'create class'
class Kls(object):
__metaclass__ = MyMeta
def __init__(self, data):
self.data = data
def printd(self):
print self.data
print 'class created'
ik = Kls('arun')
ik.printd()
ik2 = Kls('avni')
ik2.printd()
Gives us the following output:
create class
SuperMeta Called
class created
MyMeta called class '__main__.Kls' ('arun',) {}
arun
MyMeta called <class '__main__.Kls' ('avni',) {}
avni
Pythonのメタプログラミング (メタクラス)
メタクラスを簡単に説明すると，「本来コードを書かなければ実現できないような処理を黒魔術的な処理でなんとかしちゃう」ためのテクニックです。コード量を(時には劇的に)減らすことができたり，すっきりした見通しの良いクラス設計を実現できます。
JavaScriptのPrototypeのような継承が可能なPythonのクラスを作ることにしましょう。Pythonのクラス設計とJavaScriptのそれの間には異なる部分があるので，魔法を使ってこの差を埋める必要があります。
まずはコードを見てみましょう。これがPythonでJSのPrototypeを実現するメタクラスです。コードをproto.pyというファイル名で保存しておきましょう。
#!/usr/bin/env python
## -*- coding: utf-8 -*-
class PrototypeStore(dict):
""" x.prototype.XXXの値を保存するためのクラス """
def __setattr__(self, name, value):
self[name] = value
def __getattr__(self, name):
return self[name]
class PrototypeMeta(type):
""" Prototypeメタクラス(クラス生成時に呼ばれる) """
def __new__(metacls, cls_name, bases, attrs):
cls = type.__new__(metacls, cls_name, bases, attrs)
cls.prototype = PrototypeStore()
return cls
class Prototype(object):
__metaclass__ = PrototypeMeta
def __getattr__(self, name):
if name == 'prototype':
getattr(self.__class__, name)
else:
try:
getattr(object, name)
except AttributeError:
return self.__class__.prototype[name]
class TestClass(Prototype):
def __init__(self):
pass
ProtoMetaの__new__()メソッドとPrototypeの__metaclass__アトリビュートがキモ。
objectを継承した新スタイルクラスに__metaclass__というアトリビュートが設定されていると，クラスは特別な動きをします。クラス生成時に，__metaclass__のアトリビュートに設定されたクラスを起動し，__new__()メソッドを呼び出すのです。
__new__()メソッド内部を見ると，type.__new__()メソッドを呼び出してクラスを生成したあと，クラスのプロパティにprotopypeというアトリビュートを代入していることが分かります。代入されているのはProtoStoreクラスのインスタンス。ProtoStoreクラスは辞書型を継承したクラス。__setattr__()と__getattr__()を継承していて，アトリビュートの読み書きを辞書のキー操作に置き換え，任意の名前を持つアトリビュートを定義，値を保存して読み出せるオブジェクトが作れるようになっています。
type.__new__()で作ってクラスアトリビュートを割り当てられたクラスは，Prototypeクラス，およびそのサブクラスの実体として機能します。
インタラクティブシェルで「from proto import *」して，次のコードを実行してみましょう。JSのPrototypeの挙動をエミュレートした，期待通りの動きになっていることを確認します。
first = TestClass() # オブジェクトを作る
first.prototype.x = 7 # 'x'をprototypeに割り当てる
second = TestClass() # firstと同じTextClassからインスタンスを作る
print second.x # first.xと同じオブジェクトを指しているので7になる
first.x = 9 # first(インスタンス)の'x'アトリビュートに代入
print first.x # これは7でなく9を返す
del first.x # インスタンスのアトリビュートを消去
print first.x # prototype.xの返す7になるはず
禅問答
Q:Protytyoeクラスの__init__()で「self.prototype=PrototypeStore()」ってやればいいんじゃないの？
A:インスタンスごとにprototypeが定義されてしまって期待通りの動きにならない。
Q:クラスアトリビュートとして「prototype=PrototypeStore()」を定義すれば？
A:Prototypeクラス，および全てのPrototype由来のクラスで同じprototypeを共有することになり，同じく期待通りに動かない。すべてのサブクラスで「prototype=PrototypeStore()」ってやらないとならない。
Q:じゃあ__init__()で「self.__class__.prototype=PrototypeStore()」では？
A:それでもできるけど，Pythonではクラスを継承したときに__init__()が上書きされてしまうので，__init__()を定義したすべてのサブクラスで「super(XXX, self).__init__(...)」ってやらないといけない。面倒だしはまる原因になる。
JavaScriptのPrototypeのような継承が可能なPythonのクラスを作ることにしましょう。Pythonのクラス設計とJavaScriptのそれの間には異なる部分があるので，魔法を使ってこの差を埋める必要があります。
まずはコードを見てみましょう。これがPythonでJSのPrototypeを実現するメタクラスです。コードをproto.pyというファイル名で保存しておきましょう。
#!/usr/bin/env python
## -*- coding: utf-8 -*-
class PrototypeStore(dict):
""" x.prototype.XXXの値を保存するためのクラス """
def __setattr__(self, name, value):
self[name] = value
def __getattr__(self, name):
return self[name]
class PrototypeMeta(type):
""" Prototypeメタクラス(クラス生成時に呼ばれる) """
def __new__(metacls, cls_name, bases, attrs):
cls = type.__new__(metacls, cls_name, bases, attrs)
cls.prototype = PrototypeStore()
return cls
class Prototype(object):
__metaclass__ = PrototypeMeta
def __getattr__(self, name):
if name == 'prototype':
getattr(self.__class__, name)
else:
try:
getattr(object, name)
except AttributeError:
return self.__class__.prototype[name]
class TestClass(Prototype):
def __init__(self):
pass
ProtoMetaの__new__()メソッドとPrototypeの__metaclass__アトリビュートがキモ。
objectを継承した新スタイルクラスに__metaclass__というアトリビュートが設定されていると，クラスは特別な動きをします。クラス生成時に，__metaclass__のアトリビュートに設定されたクラスを起動し，__new__()メソッドを呼び出すのです。
__new__()メソッド内部を見ると，type.__new__()メソッドを呼び出してクラスを生成したあと，クラスのプロパティにprotopypeというアトリビュートを代入していることが分かります。代入されているのはProtoStoreクラスのインスタンス。ProtoStoreクラスは辞書型を継承したクラス。__setattr__()と__getattr__()を継承していて，アトリビュートの読み書きを辞書のキー操作に置き換え，任意の名前を持つアトリビュートを定義，値を保存して読み出せるオブジェクトが作れるようになっています。
type.__new__()で作ってクラスアトリビュートを割り当てられたクラスは，Prototypeクラス，およびそのサブクラスの実体として機能します。
インタラクティブシェルで「from proto import *」して，次のコードを実行してみましょう。JSのPrototypeの挙動をエミュレートした，期待通りの動きになっていることを確認します。
first = TestClass() # オブジェクトを作る
first.prototype.x = 7 # 'x'をprototypeに割り当てる
second = TestClass() # firstと同じTextClassからインスタンスを作る
print second.x # first.xと同じオブジェクトを指しているので7になる
first.x = 9 # first(インスタンス)の'x'アトリビュートに代入
print first.x # これは7でなく9を返す
del first.x # インスタンスのアトリビュートを消去
print first.x # prototype.xの返す7になるはず
禅問答
Q:Protytyoeクラスの__init__()で「self.prototype=PrototypeStore()」ってやればいいんじゃないの？
A:インスタンスごとにprototypeが定義されてしまって期待通りの動きにならない。
Q:クラスアトリビュートとして「prototype=PrototypeStore()」を定義すれば？
A:Prototypeクラス，および全てのPrototype由来のクラスで同じprototypeを共有することになり，同じく期待通りに動かない。すべてのサブクラスで「prototype=PrototypeStore()」ってやらないとならない。
Q:じゃあ__init__()で「self.__class__.prototype=PrototypeStore()」では？
A:それでもできるけど，Pythonではクラスを継承したときに__init__()が上書きされてしまうので，__init__()を定義したすべてのサブクラスで「super(XXX, self).__init__(...)」ってやらないといけない。面倒だしはまる原因になる。