1'''"Executable documentation" for the pickle module. 2 3Extensive comments about the pickle protocols and pickle-machine opcodes 4can be found here. Some functions meant for external use: 5 6genops(pickle) 7 Generate all the opcodes in a pickle, as (opcode, arg, position) triples. 8 9dis(pickle, out=None, memo=None, indentlevel=4) 10 Print a symbolic disassembly of a pickle. 11''' 12 13__all__ = ['dis', 'genops', 'optimize'] 14 15# Other ideas: 16# 17# - A pickle verifier: read a pickle and check it exhaustively for 18# well-formedness. dis() does a lot of this already. 19# 20# - A protocol identifier: examine a pickle and return its protocol number 21# (== the highest .proto attr value among all the opcodes in the pickle). 22# dis() already prints this info at the end. 23# 24# - A pickle optimizer: for example, tuple-building code is sometimes more 25# elaborate than necessary, catering for the possibility that the tuple 26# is recursive. Or lots of times a PUT is generated that's never accessed 27# by a later GET. 28 29 30""" 31"A pickle" is a program for a virtual pickle machine (PM, but more accurately 32called an unpickling machine). It's a sequence of opcodes, interpreted by the 33PM, building an arbitrarily complex Python object. 34 35For the most part, the PM is very simple: there are no looping, testing, or 36conditional instructions, no arithmetic and no function calls. Opcodes are 37executed once each, from first to last, until a STOP opcode is reached. 38 39The PM has two data areas, "the stack" and "the memo". 40 41Many opcodes push Python objects onto the stack; e.g., INT pushes a Python 42integer object on the stack, whose value is gotten from a decimal string 43literal immediately following the INT opcode in the pickle bytestream. Other 44opcodes take Python objects off the stack. The result of unpickling is 45whatever object is left on the stack when the final STOP opcode is executed. 46 47The memo is simply an array of objects, or it can be implemented as a dict 48mapping little integers to objects. The memo serves as the PM's "long term 49memory", and the little integers indexing the memo are akin to variable 50names. Some opcodes pop a stack object into the memo at a given index, 51and others push a memo object at a given index onto the stack again. 52 53At heart, that's all the PM has. Subtleties arise for these reasons: 54 55+ Object identity. Objects can be arbitrarily complex, and subobjects 56 may be shared (for example, the list [a, a] refers to the same object a 57 twice). It can be vital that unpickling recreate an isomorphic object 58 graph, faithfully reproducing sharing. 59 60+ Recursive objects. For example, after "L = []; L.append(L)", L is a 61 list, and L[0] is the same list. This is related to the object identity 62 point, and some sequences of pickle opcodes are subtle in order to 63 get the right result in all cases. 64 65+ Things pickle doesn't know everything about. Examples of things pickle 66 does know everything about are Python's builtin scalar and container 67 types, like ints and tuples. They generally have opcodes dedicated to 68 them. For things like module references and instances of user-defined 69 classes, pickle's knowledge is limited. Historically, many enhancements 70 have been made to the pickle protocol in order to do a better (faster, 71 and/or more compact) job on those. 72 73+ Backward compatibility and micro-optimization. As explained below, 74 pickle opcodes never go away, not even when better ways to do a thing 75 get invented. The repertoire of the PM just keeps growing over time. 76 For example, protocol 0 had two opcodes for building Python integers (INT 77 and LONG), protocol 1 added three more for more-efficient pickling of short 78 integers, and protocol 2 added two more for more-efficient pickling of 79 long integers (before protocol 2, the only ways to pickle a Python long 80 took time quadratic in the number of digits, for both pickling and 81 unpickling). "Opcode bloat" isn't so much a subtlety as a source of 82 wearying complication. 83 84 85Pickle protocols: 86 87For compatibility, the meaning of a pickle opcode never changes. Instead new 88pickle opcodes get added, and each version's unpickler can handle all the 89pickle opcodes in all protocol versions to date. So old pickles continue to 90be readable forever. The pickler can generally be told to restrict itself to 91the subset of opcodes available under previous protocol versions too, so that 92users can create pickles under the current version readable by older 93versions. However, a pickle does not contain its version number embedded 94within it. If an older unpickler tries to read a pickle using a later 95protocol, the result is most likely an exception due to seeing an unknown (in 96the older unpickler) opcode. 97 98The original pickle used what's now called "protocol 0", and what was called 99"text mode" before Python 2.3. The entire pickle bytestream is made up of 100printable 7-bit ASCII characters, plus the newline character, in protocol 0. 101That's why it was called text mode. Protocol 0 is small and elegant, but 102sometimes painfully inefficient. 103 104The second major set of additions is now called "protocol 1", and was called 105"binary mode" before Python 2.3. This added many opcodes with arguments 106consisting of arbitrary bytes, including NUL bytes and unprintable "high bit" 107bytes. Binary mode pickles can be substantially smaller than equivalent 108text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte 109int as 4 bytes following the opcode, which is cheaper to unpickle than the 110(perhaps) 11-character decimal string attached to INT. Protocol 1 also added 111a number of opcodes that operate on many stack elements at once (like APPENDS 112and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE). 113 114The third major set of additions came in Python 2.3, and is called "protocol 1152". This added: 116 117- A better way to pickle instances of new-style classes (NEWOBJ). 118 119- A way for a pickle to identify its protocol (PROTO). 120 121- Time- and space- efficient pickling of long ints (LONG{1,4}). 122 123- Shortcuts for small tuples (TUPLE{1,2,3}}. 124 125- Dedicated opcodes for bools (NEWTRUE, NEWFALSE). 126 127- The "extension registry", a vector of popular objects that can be pushed 128 efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but 129 the registry contents are predefined (there's nothing akin to the memo's 130 PUT). 131 132Another independent change with Python 2.3 is the abandonment of any 133pretense that it might be safe to load pickles received from untrusted 134parties -- no sufficient security analysis has been done to guarantee 135this and there isn't a use case that warrants the expense of such an 136analysis. 137 138To this end, all tests for __safe_for_unpickling__ or for 139copy_reg.safe_constructors are removed from the unpickling code. 140References to these variables in the descriptions below are to be seen 141as describing unpickling in Python 2.2 and before. 142""" 143 144# Meta-rule: Descriptions are stored in instances of descriptor objects, 145# with plain constructors. No meta-language is defined from which 146# descriptors could be constructed. If you want, e.g., XML, write a little 147# program to generate XML from the objects. 148 149############################################################################## 150# Some pickle opcodes have an argument, following the opcode in the 151# bytestream. An argument is of a specific type, described by an instance 152# of ArgumentDescriptor. These are not to be confused with arguments taken 153# off the stack -- ArgumentDescriptor applies only to arguments embedded in 154# the opcode stream, immediately following an opcode. 155 156# Represents the number of bytes consumed by an argument delimited by the 157# next newline character. 158UP_TO_NEWLINE = -1 159 160# Represents the number of bytes consumed by a two-argument opcode where 161# the first argument gives the number of bytes in the second argument. 162TAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int 163TAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int 164 165class ArgumentDescriptor(object): 166 __slots__ = ( 167 # name of descriptor record, also a module global name; a string 168 'name', 169 170 # length of argument, in bytes; an int; UP_TO_NEWLINE and 171 # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length 172 # cases 173 'n', 174 175 # a function taking a file-like object, reading this kind of argument 176 # from the object at the current position, advancing the current 177 # position by n bytes, and returning the value of the argument 178 'reader', 179 180 # human-readable docs for this arg descriptor; a string 181 'doc', 182 ) 183 184 def __init__(self, name, n, reader, doc): 185 assert isinstance(name, str) 186 self.name = name 187 188 assert isinstance(n, int) and (n >= 0 or 189 n in (UP_TO_NEWLINE, 190 TAKEN_FROM_ARGUMENT1, 191 TAKEN_FROM_ARGUMENT4)) 192 self.n = n 193 194 self.reader = reader 195 196 assert isinstance(doc, str) 197 self.doc = doc 198 199from struct import unpack as _unpack 200 201def read_uint1(f): 202 r""" 203 >>> import StringIO 204 >>> read_uint1(StringIO.StringIO('\xff')) 205 255 206 """ 207 208 data = f.read(1) 209 if data: 210 return ord(data) 211 raise ValueError("not enough data in stream to read uint1") 212 213uint1 = ArgumentDescriptor( 214 name='uint1', 215 n=1, 216 reader=read_uint1, 217 doc="One-byte unsigned integer.") 218 219 220def read_uint2(f): 221 r""" 222 >>> import StringIO 223 >>> read_uint2(StringIO.StringIO('\xff\x00')) 224 255 225 >>> read_uint2(StringIO.StringIO('\xff\xff')) 226 65535 227 """ 228 229 data = f.read(2) 230 if len(data) == 2: 231 return _unpack("<H", data)[0] 232 raise ValueError("not enough data in stream to read uint2") 233 234uint2 = ArgumentDescriptor( 235 name='uint2', 236 n=2, 237 reader=read_uint2, 238 doc="Two-byte unsigned integer, little-endian.") 239 240 241def read_int4(f): 242 r""" 243 >>> import StringIO 244 >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00')) 245 255 246 >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31) 247 True 248 """ 249 250 data = f.read(4) 251 if len(data) == 4: 252 return _unpack("<i", data)[0] 253 raise ValueError("not enough data in stream to read int4") 254 255int4 = ArgumentDescriptor( 256 name='int4', 257 n=4, 258 reader=read_int4, 259 doc="Four-byte signed integer, little-endian, 2's complement.") 260 261 262def read_stringnl(f, decode=True, stripquotes=True): 263 r""" 264 >>> import StringIO 265 >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n")) 266 'abcd' 267 268 >>> read_stringnl(StringIO.StringIO("\n")) 269 Traceback (most recent call last): 270 ... 271 ValueError: no string quotes around '' 272 273 >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False) 274 '' 275 276 >>> read_stringnl(StringIO.StringIO("''\n")) 277 '' 278 279 >>> read_stringnl(StringIO.StringIO('"abcd"')) 280 Traceback (most recent call last): 281 ... 282 ValueError: no newline found when trying to read stringnl 283 284 Embedded escapes are undone in the result. 285 >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'")) 286 'a\n\\b\x00c\td' 287 """ 288 289 data = f.readline() 290 if not data.endswith('\n'): 291 raise ValueError("no newline found when trying to read stringnl") 292 data = data[:-1] # lose the newline 293 294 if stripquotes: 295 for q in "'\"": 296 if data.startswith(q): 297 if not data.endswith(q): 298 raise ValueError("strinq quote %r not found at both " 299 "ends of %r" % (q, data)) 300 data = data[1:-1] 301 break 302 else: 303 raise ValueError("no string quotes around %r" % data) 304 305 # I'm not sure when 'string_escape' was added to the std codecs; it's 306 # crazy not to use it if it's there. 307 if decode: 308 data = data.decode('string_escape') 309 return data 310 311stringnl = ArgumentDescriptor( 312 name='stringnl', 313 n=UP_TO_NEWLINE, 314 reader=read_stringnl, 315 doc="""A newline-terminated string. 316 317 This is a repr-style string, with embedded escapes, and 318 bracketing quotes. 319 """) 320 321def read_stringnl_noescape(f): 322 return read_stringnl(f, decode=False, stripquotes=False) 323 324stringnl_noescape = ArgumentDescriptor( 325 name='stringnl_noescape', 326 n=UP_TO_NEWLINE, 327 reader=read_stringnl_noescape, 328 doc="""A newline-terminated string. 329 330 This is a str-style string, without embedded escapes, 331 or bracketing quotes. It should consist solely of 332 printable ASCII characters. 333 """) 334 335def read_stringnl_noescape_pair(f): 336 r""" 337 >>> import StringIO 338 >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk")) 339 'Queue Empty' 340 """ 341 342 return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) 343 344stringnl_noescape_pair = ArgumentDescriptor( 345 name='stringnl_noescape_pair', 346 n=UP_TO_NEWLINE, 347 reader=read_stringnl_noescape_pair, 348 doc="""A pair of newline-terminated strings. 349 350 These are str-style strings, without embedded 351 escapes, or bracketing quotes. They should 352 consist solely of printable ASCII characters. 353 The pair is returned as a single string, with 354 a single blank separating the two strings. 355 """) 356 357def read_string4(f): 358 r""" 359 >>> import StringIO 360 >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc")) 361 '' 362 >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef")) 363 'abc' 364 >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef")) 365 Traceback (most recent call last): 366 ... 367 ValueError: expected 50331648 bytes in a string4, but only 6 remain 368 """ 369 370 n = read_int4(f) 371 if n < 0: 372 raise ValueError("string4 byte count < 0: %d" % n) 373 data = f.read(n) 374 if len(data) == n: 375 return data 376 raise ValueError("expected %d bytes in a string4, but only %d remain" % 377 (n, len(data))) 378 379string4 = ArgumentDescriptor( 380 name="string4", 381 n=TAKEN_FROM_ARGUMENT4, 382 reader=read_string4, 383 doc="""A counted string. 384 385 The first argument is a 4-byte little-endian signed int giving 386 the number of bytes in the string, and the second argument is 387 that many bytes. 388 """) 389 390 391def read_string1(f): 392 r""" 393 >>> import StringIO 394 >>> read_string1(StringIO.StringIO("\x00")) 395 '' 396 >>> read_string1(StringIO.StringIO("\x03abcdef")) 397 'abc' 398 """ 399 400 n = read_uint1(f) 401 assert n >= 0 402 data = f.read(n) 403 if len(data) == n: 404 return data 405 raise ValueError("expected %d bytes in a string1, but only %d remain" % 406 (n, len(data))) 407 408string1 = ArgumentDescriptor( 409 name="string1", 410 n=TAKEN_FROM_ARGUMENT1, 411 reader=read_string1, 412 doc="""A counted string. 413 414 The first argument is a 1-byte unsigned int giving the number 415 of bytes in the string, and the second argument is that many 416 bytes. 417 """) 418 419 420def read_unicodestringnl(f): 421 r""" 422 >>> import StringIO 423 >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk")) 424 u'abc\uabcd' 425 """ 426 427 data = f.readline() 428 if not data.endswith('\n'): 429 raise ValueError("no newline found when trying to read " 430 "unicodestringnl") 431 data = data[:-1] # lose the newline 432 return unicode(data, 'raw-unicode-escape') 433 434unicodestringnl = ArgumentDescriptor( 435 name='unicodestringnl', 436 n=UP_TO_NEWLINE, 437 reader=read_unicodestringnl, 438 doc="""A newline-terminated Unicode string. 439 440 This is raw-unicode-escape encoded, so consists of 441 printable ASCII characters, and may contain embedded 442 escape sequences. 443 """) 444 445def read_unicodestring4(f): 446 r""" 447 >>> import StringIO 448 >>> s = u'abcd\uabcd' 449 >>> enc = s.encode('utf-8') 450 >>> enc 451 'abcd\xea\xaf\x8d' 452 >>> n = chr(len(enc)) + chr(0) * 3 # little-endian 4-byte length 453 >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk')) 454 >>> s == t 455 True 456 457 >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1])) 458 Traceback (most recent call last): 459 ... 460 ValueError: expected 7 bytes in a unicodestring4, but only 6 remain 461 """ 462 463 n = read_int4(f) 464 if n < 0: 465 raise ValueError("unicodestring4 byte count < 0: %d" % n) 466 data = f.read(n) 467 if len(data) == n: 468 return unicode(data, 'utf-8') 469 raise ValueError("expected %d bytes in a unicodestring4, but only %d " 470 "remain" % (n, len(data))) 471 472unicodestring4 = ArgumentDescriptor( 473 name="unicodestring4", 474 n=TAKEN_FROM_ARGUMENT4, 475 reader=read_unicodestring4, 476 doc="""A counted Unicode string. 477 478 The first argument is a 4-byte little-endian signed int 479 giving the number of bytes in the string, and the second 480 argument-- the UTF-8 encoding of the Unicode string -- 481 contains that many bytes. 482 """) 483 484 485def read_decimalnl_short(f): 486 r""" 487 >>> import StringIO 488 >>> read_decimalnl_short(StringIO.StringIO("1234\n56")) 489 1234 490 491 >>> read_decimalnl_short(StringIO.StringIO("1234L\n56")) 492 Traceback (most recent call last): 493 ... 494 ValueError: trailing 'L' not allowed in '1234L' 495 """ 496 497 s = read_stringnl(f, decode=False, stripquotes=False) 498 if s.endswith("L"): 499 raise ValueError("trailing 'L' not allowed in %r" % s) 500 501 # It's not necessarily true that the result fits in a Python short int: 502 # the pickle may have been written on a 64-bit box. There's also a hack 503 # for True and False here. 504 if s == "00": 505 return False 506 elif s == "01": 507 return True 508 509 try: 510 return int(s) 511 except OverflowError: 512 return long(s) 513 514def read_decimalnl_long(f): 515 r""" 516 >>> import StringIO 517 518 >>> read_decimalnl_long(StringIO.StringIO("1234\n56")) 519 Traceback (most recent call last): 520 ... 521 ValueError: trailing 'L' required in '1234' 522 523 Someday the trailing 'L' will probably go away from this output. 524 525 >>> read_decimalnl_long(StringIO.StringIO("1234L\n56")) 526 1234L 527 528 >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\n6")) 529 123456789012345678901234L 530 """ 531 532 s = read_stringnl(f, decode=False, stripquotes=False) 533 if not s.endswith("L"): 534 raise ValueError("trailing 'L' required in %r" % s) 535 return long(s) 536 537 538decimalnl_short = ArgumentDescriptor( 539 name='decimalnl_short', 540 n=UP_TO_NEWLINE, 541 reader=read_decimalnl_short, 542 doc="""A newline-terminated decimal integer literal. 543 544 This never has a trailing 'L', and the integer fit 545 in a short Python int on the box where the pickle 546 was written -- but there's no guarantee it will fit 547 in a short Python int on the box where the pickle 548 is read. 549 """) 550 551decimalnl_long = ArgumentDescriptor( 552 name='decimalnl_long', 553 n=UP_TO_NEWLINE, 554 reader=read_decimalnl_long, 555 doc="""A newline-terminated decimal integer literal. 556 557 This has a trailing 'L', and can represent integers 558 of any size. 559 """) 560 561 562def read_floatnl(f): 563 r""" 564 >>> import StringIO 565 >>> read_floatnl(StringIO.StringIO("-1.25\n6")) 566 -1.25 567 """ 568 s = read_stringnl(f, decode=False, stripquotes=False) 569 return float(s) 570 571floatnl = ArgumentDescriptor( 572 name='floatnl', 573 n=UP_TO_NEWLINE, 574 reader=read_floatnl, 575 doc="""A newline-terminated decimal floating literal. 576 577 In general this requires 17 significant digits for roundtrip 578 identity, and pickling then unpickling infinities, NaNs, and 579 minus zero doesn't work across boxes, or on some boxes even 580 on itself (e.g., Windows can't read the strings it produces 581 for infinities or NaNs). 582 """) 583 584def read_float8(f): 585 r""" 586 >>> import StringIO, struct 587 >>> raw = struct.pack(">d", -1.25) 588 >>> raw 589 '\xbf\xf4\x00\x00\x00\x00\x00\x00' 590 >>> read_float8(StringIO.StringIO(raw + "\n")) 591 -1.25 592 """ 593 594 data = f.read(8) 595 if len(data) == 8: 596 return _unpack(">d", data)[0] 597 raise ValueError("not enough data in stream to read float8") 598 599 600float8 = ArgumentDescriptor( 601 name='float8', 602 n=8, 603 reader=read_float8, 604 doc="""An 8-byte binary representation of a float, big-endian. 605 606 The format is unique to Python, and shared with the struct 607 module (format string '>d') "in theory" (the struct and cPickle 608 implementations don't share the code -- they should). It's 609 strongly related to the IEEE-754 double format, and, in normal 610 cases, is in fact identical to the big-endian 754 double format. 611 On other boxes the dynamic range is limited to that of a 754 612 double, and "add a half and chop" rounding is used to reduce 613 the precision to 53 bits. However, even on a 754 box, 614 infinities, NaNs, and minus zero may not be handled correctly 615 (may not survive roundtrip pickling intact). 616 """) 617 618# Protocol 2 formats 619 620from pickle import decode_long 621 622def read_long1(f): 623 r""" 624 >>> import StringIO 625 >>> read_long1(StringIO.StringIO("\x00")) 626 0L 627 >>> read_long1(StringIO.StringIO("\x02\xff\x00")) 628 255L 629 >>> read_long1(StringIO.StringIO("\x02\xff\x7f")) 630 32767L 631 >>> read_long1(StringIO.StringIO("\x02\x00\xff")) 632 -256L 633 >>> read_long1(StringIO.StringIO("\x02\x00\x80")) 634 -32768L 635 """ 636 637 n = read_uint1(f) 638 data = f.read(n) 639 if len(data) != n: 640 raise ValueError("not enough data in stream to read long1") 641 return decode_long(data) 642 643long1 = ArgumentDescriptor( 644 name="long1", 645 n=TAKEN_FROM_ARGUMENT1, 646 reader=read_long1, 647 doc="""A binary long, little-endian, using 1-byte size. 648 649 This first reads one byte as an unsigned size, then reads that 650 many bytes and interprets them as a little-endian 2's-complement long. 651 If the size is 0, that's taken as a shortcut for the long 0L. 652 """) 653 654def read_long4(f): 655 r""" 656 >>> import StringIO 657 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x00")) 658 255L 659 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x7f")) 660 32767L 661 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\xff")) 662 -256L 663 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\x80")) 664 -32768L 665 >>> read_long1(StringIO.StringIO("\x00\x00\x00\x00")) 666 0L 667 """ 668 669 n = read_int4(f) 670 if n < 0: 671 raise ValueError("long4 byte count < 0: %d" % n) 672 data = f.read(n) 673 if len(data) != n: 674 raise ValueError("not enough data in stream to read long4") 675 return decode_long(data) 676 677long4 = ArgumentDescriptor( 678 name="long4", 679 n=TAKEN_FROM_ARGUMENT4, 680 reader=read_long4, 681 doc="""A binary representation of a long, little-endian. 682 683 This first reads four bytes as a signed size (but requires the 684 size to be >= 0), then reads that many bytes and interprets them 685 as a little-endian 2's-complement long. If the size is 0, that's taken 686 as a shortcut for the long 0L, although LONG1 should really be used 687 then instead (and in any case where # of bytes < 256). 688 """) 689 690 691############################################################################## 692# Object descriptors. The stack used by the pickle machine holds objects, 693# and in the stack_before and stack_after attributes of OpcodeInfo 694# descriptors we need names to describe the various types of objects that can 695# appear on the stack. 696 697class StackObject(object): 698 __slots__ = ( 699 # name of descriptor record, for info only 700 'name', 701 702 # type of object, or tuple of type objects (meaning the object can 703 # be of any type in the tuple) 704 'obtype', 705 706 # human-readable docs for this kind of stack object; a string 707 'doc', 708 ) 709 710 def __init__(self, name, obtype, doc): 711 assert isinstance(name, str) 712 self.name = name 713 714 assert isinstance(obtype, type) or isinstance(obtype, tuple) 715 if isinstance(obtype, tuple): 716 for contained in obtype: 717 assert isinstance(contained, type) 718 self.obtype = obtype 719 720 assert isinstance(doc, str) 721 self.doc = doc 722 723 def __repr__(self): 724 return self.name 725 726 727pyint = StackObject( 728 name='int', 729 obtype=int, 730 doc="A short (as opposed to long) Python integer object.") 731 732pylong = StackObject( 733 name='long', 734 obtype=long, 735 doc="A long (as opposed to short) Python integer object.") 736 737pyinteger_or_bool = StackObject( 738 name='int_or_bool', 739 obtype=(int, long, bool), 740 doc="A Python integer object (short or long), or " 741 "a Python bool.") 742 743pybool = StackObject( 744 name='bool', 745 obtype=(bool,), 746 doc="A Python bool object.") 747 748pyfloat = StackObject( 749 name='float', 750 obtype=float, 751 doc="A Python float object.") 752 753pystring = StackObject( 754 name='str', 755 obtype=str, 756 doc="A Python string object.") 757 758pyunicode = StackObject( 759 name='unicode', 760 obtype=unicode, 761 doc="A Python Unicode string object.") 762 763pynone = StackObject( 764 name="None", 765 obtype=type(None), 766 doc="The Python None object.") 767 768pytuple = StackObject( 769 name="tuple", 770 obtype=tuple, 771 doc="A Python tuple object.") 772 773pylist = StackObject( 774 name="list", 775 obtype=list, 776 doc="A Python list object.") 777 778pydict = StackObject( 779 name="dict", 780 obtype=dict, 781 doc="A Python dict object.") 782 783anyobject = StackObject( 784 name='any', 785 obtype=object, 786 doc="Any kind of object whatsoever.") 787 788markobject = StackObject( 789 name="mark", 790 obtype=StackObject, 791 doc="""'The mark' is a unique object. 792 793 Opcodes that operate on a variable number of objects 794 generally don't embed the count of objects in the opcode, 795 or pull it off the stack. Instead the MARK opcode is used 796 to push a special marker object on the stack, and then 797 some other opcodes grab all the objects from the top of 798 the stack down to (but not including) the topmost marker 799 object. 800 """) 801 802stackslice = StackObject( 803 name="stackslice", 804 obtype=StackObject, 805 doc="""An object representing a contiguous slice of the stack. 806 807 This is used in conjuction with markobject, to represent all 808 of the stack following the topmost markobject. For example, 809 the POP_MARK opcode changes the stack from 810 811 [..., markobject, stackslice] 812 to 813 [...] 814 815 No matter how many object are on the stack after the topmost 816 markobject, POP_MARK gets rid of all of them (including the 817 topmost markobject too). 818 """) 819 820############################################################################## 821# Descriptors for pickle opcodes. 822 823class OpcodeInfo(object): 824 825 __slots__ = ( 826 # symbolic name of opcode; a string 827 'name', 828 829 # the code used in a bytestream to represent the opcode; a 830 # one-character string 831 'code', 832 833 # If the opcode has an argument embedded in the byte string, an 834 # instance of ArgumentDescriptor specifying its type. Note that 835 # arg.reader(s) can be used to read and decode the argument from 836 # the bytestream s, and arg.doc documents the format of the raw 837 # argument bytes. If the opcode doesn't have an argument embedded 838 # in the bytestream, arg should be None. 839 'arg', 840 841 # what the stack looks like before this opcode runs; a list 842 'stack_before', 843 844 # what the stack looks like after this opcode runs; a list 845 'stack_after', 846 847 # the protocol number in which this opcode was introduced; an int 848 'proto', 849 850 # human-readable docs for this opcode; a string 851 'doc', 852 ) 853 854 def __init__(self, name, code, arg, 855 stack_before, stack_after, proto, doc): 856 assert isinstance(name, str) 857 self.name = name 858 859 assert isinstance(code, str) 860 assert len(code) == 1 861 self.code = code 862 863 assert arg is None or isinstance(arg, ArgumentDescriptor) 864 self.arg = arg 865 866 assert isinstance(stack_before, list) 867 for x in stack_before: 868 assert isinstance(x, StackObject) 869 self.stack_before = stack_before 870 871 assert isinstance(stack_after, list) 872 for x in stack_after: 873 assert isinstance(x, StackObject) 874 self.stack_after = stack_after 875 876 assert isinstance(proto, int) and 0 <= proto <= 2 877 self.proto = proto 878 879 assert isinstance(doc, str) 880 self.doc = doc 881 882I = OpcodeInfo 883opcodes = [ 884 885 # Ways to spell integers. 886 887 I(name='INT', 888 code='I', 889 arg=decimalnl_short, 890 stack_before=[], 891 stack_after=[pyinteger_or_bool], 892 proto=0, 893 doc="""Push an integer or bool. 894 895 The argument is a newline-terminated decimal literal string. 896 897 The intent may have been that this always fit in a short Python int, 898 but INT can be generated in pickles written on a 64-bit box that 899 require a Python long on a 32-bit box. The difference between this 900 and LONG then is that INT skips a trailing 'L', and produces a short 901 int whenever possible. 902 903 Another difference is due to that, when bool was introduced as a 904 distinct type in 2.3, builtin names True and False were also added to 905 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, 906 True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". 907 Leading zeroes are never produced for a genuine integer. The 2.3 908 (and later) unpicklers special-case these and return bool instead; 909 earlier unpicklers ignore the leading "0" and return the int. 910 """), 911 912 I(name='BININT', 913 code='J', 914 arg=int4, 915 stack_before=[], 916 stack_after=[pyint], 917 proto=1, 918 doc="""Push a four-byte signed integer. 919 920 This handles the full range of Python (short) integers on a 32-bit 921 box, directly as binary bytes (1 for the opcode and 4 for the integer). 922 If the integer is non-negative and fits in 1 or 2 bytes, pickling via 923 BININT1 or BININT2 saves space. 924 """), 925 926 I(name='BININT1', 927 code='K', 928 arg=uint1, 929 stack_before=[], 930 stack_after=[pyint], 931 proto=1, 932 doc="""Push a one-byte unsigned integer. 933 934 This is a space optimization for pickling very small non-negative ints, 935 in range(256). 936 """), 937 938 I(name='BININT2', 939 code='M', 940 arg=uint2, 941 stack_before=[], 942 stack_after=[pyint], 943 proto=1, 944 doc="""Push a two-byte unsigned integer. 945 946 This is a space optimization for pickling small positive ints, in 947 range(256, 2**16). Integers in range(256) can also be pickled via 948 BININT2, but BININT1 instead saves a byte. 949 """), 950 951 I(name='LONG', 952 code='L', 953 arg=decimalnl_long, 954 stack_before=[], 955 stack_after=[pylong], 956 proto=0, 957 doc="""Push a long integer. 958 959 The same as INT, except that the literal ends with 'L', and always 960 unpickles to a Python long. There doesn't seem a real purpose to the 961 trailing 'L'. 962 963 Note that LONG takes time quadratic in the number of digits when 964 unpickling (this is simply due to the nature of decimal->binary 965 conversion). Proto 2 added linear-time (in C; still quadratic-time 966 in Python) LONG1 and LONG4 opcodes. 967 """), 968 969 I(name="LONG1", 970 code='\x8a', 971 arg=long1, 972 stack_before=[], 973 stack_after=[pylong], 974 proto=2, 975 doc="""Long integer using one-byte length. 976 977 A more efficient encoding of a Python long; the long1 encoding 978 says it all."""), 979 980 I(name="LONG4", 981 code='\x8b', 982 arg=long4, 983 stack_before=[], 984 stack_after=[pylong], 985 proto=2, 986 doc="""Long integer using found-byte length. 987 988 A more efficient encoding of a Python long; the long4 encoding 989 says it all."""), 990 991 # Ways to spell strings (8-bit, not Unicode). 992 993 I(name='STRING', 994 code='S', 995 arg=stringnl, 996 stack_before=[], 997 stack_after=[pystring], 998 proto=0, 999 doc="""Push a Python string object. 1000 1001 The argument is a repr-style string, with bracketing quote characters, 1002 and perhaps embedded escapes. The argument extends until the next 1003 newline character. 1004 """), 1005 1006 I(name='BINSTRING', 1007 code='T', 1008 arg=string4, 1009 stack_before=[], 1010 stack_after=[pystring], 1011 proto=1, 1012 doc="""Push a Python string object. 1013 1014 There are two arguments: the first is a 4-byte little-endian signed int 1015 giving the number of bytes in the string, and the second is that many 1016 bytes, which are taken literally as the string content. 1017 """), 1018 1019 I(name='SHORT_BINSTRING', 1020 code='U', 1021 arg=string1, 1022 stack_before=[], 1023 stack_after=[pystring], 1024 proto=1, 1025 doc="""Push a Python string object. 1026 1027 There are two arguments: the first is a 1-byte unsigned int giving 1028 the number of bytes in the string, and the second is that many bytes, 1029 which are taken literally as the string content. 1030 """), 1031 1032 # Ways to spell None. 1033 1034 I(name='NONE', 1035 code='N', 1036 arg=None, 1037 stack_before=[], 1038 stack_after=[pynone], 1039 proto=0, 1040 doc="Push None on the stack."), 1041 1042 # Ways to spell bools, starting with proto 2. See INT for how this was 1043 # done before proto 2. 1044 1045 I(name='NEWTRUE', 1046 code='\x88', 1047 arg=None, 1048 stack_before=[], 1049 stack_after=[pybool], 1050 proto=2, 1051 doc="""True. 1052 1053 Push True onto the stack."""), 1054 1055 I(name='NEWFALSE', 1056 code='\x89', 1057 arg=None, 1058 stack_before=[], 1059 stack_after=[pybool], 1060 proto=2, 1061 doc="""True. 1062 1063 Push False onto the stack."""), 1064 1065 # Ways to spell Unicode strings. 1066 1067 I(name='UNICODE', 1068 code='V', 1069 arg=unicodestringnl, 1070 stack_before=[], 1071 stack_after=[pyunicode], 1072 proto=0, # this may be pure-text, but it's a later addition 1073 doc="""Push a Python Unicode string object. 1074 1075 The argument is a raw-unicode-escape encoding of a Unicode string, 1076 and so may contain embedded escape sequences. The argument extends 1077 until the next newline character. 1078 """), 1079 1080 I(name='BINUNICODE', 1081 code='X', 1082 arg=unicodestring4, 1083 stack_before=[], 1084 stack_after=[pyunicode], 1085 proto=1, 1086 doc="""Push a Python Unicode string object. 1087 1088 There are two arguments: the first is a 4-byte little-endian signed int 1089 giving the number of bytes in the string. The second is that many 1090 bytes, and is the UTF-8 encoding of the Unicode string. 1091 """), 1092 1093 # Ways to spell floats. 1094 1095 I(name='FLOAT', 1096 code='F', 1097 arg=floatnl, 1098 stack_before=[], 1099 stack_after=[pyfloat], 1100 proto=0, 1101 doc="""Newline-terminated decimal float literal. 1102 1103 The argument is repr(a_float), and in general requires 17 significant 1104 digits for roundtrip conversion to be an identity (this is so for 1105 IEEE-754 double precision values, which is what Python float maps to 1106 on most boxes). 1107 1108 In general, FLOAT cannot be used to transport infinities, NaNs, or 1109 minus zero across boxes (or even on a single box, if the platform C 1110 library can't read the strings it produces for such things -- Windows 1111 is like that), but may do less damage than BINFLOAT on boxes with 1112 greater precision or dynamic range than IEEE-754 double. 1113 """), 1114 1115 I(name='BINFLOAT', 1116 code='G', 1117 arg=float8, 1118 stack_before=[], 1119 stack_after=[pyfloat], 1120 proto=1, 1121 doc="""Float stored in binary form, with 8 bytes of data. 1122 1123 This generally requires less than half the space of FLOAT encoding. 1124 In general, BINFLOAT cannot be used to transport infinities, NaNs, or 1125 minus zero, raises an exception if the exponent exceeds the range of 1126 an IEEE-754 double, and retains no more than 53 bits of precision (if 1127 there are more than that, "add a half and chop" rounding is used to 1128 cut it back to 53 significant bits). 1129 """), 1130 1131 # Ways to build lists. 1132 1133 I(name='EMPTY_LIST', 1134 code=']', 1135 arg=None, 1136 stack_before=[], 1137 stack_after=[pylist], 1138 proto=1, 1139 doc="Push an empty list."), 1140 1141 I(name='APPEND', 1142 code='a', 1143 arg=None, 1144 stack_before=[pylist, anyobject], 1145 stack_after=[pylist], 1146 proto=0, 1147 doc="""Append an object to a list. 1148 1149 Stack before: ... pylist anyobject 1150 Stack after: ... pylist+[anyobject] 1151 1152 although pylist is really extended in-place. 1153 """), 1154 1155 I(name='APPENDS', 1156 code='e', 1157 arg=None, 1158 stack_before=[pylist, markobject, stackslice], 1159 stack_after=[pylist], 1160 proto=1, 1161 doc="""Extend a list by a slice of stack objects. 1162 1163 Stack before: ... pylist markobject stackslice 1164 Stack after: ... pylist+stackslice 1165 1166 although pylist is really extended in-place. 1167 """), 1168 1169 I(name='LIST', 1170 code='l', 1171 arg=None, 1172 stack_before=[markobject, stackslice], 1173 stack_after=[pylist], 1174 proto=0, 1175 doc="""Build a list out of the topmost stack slice, after markobject. 1176 1177 All the stack entries following the topmost markobject are placed into 1178 a single Python list, which single list object replaces all of the 1179 stack from the topmost markobject onward. For example, 1180 1181 Stack before: ... markobject 1 2 3 'abc' 1182 Stack after: ... [1, 2, 3, 'abc'] 1183 """), 1184 1185 # Ways to build tuples. 1186 1187 I(name='EMPTY_TUPLE', 1188 code=')', 1189 arg=None, 1190 stack_before=[], 1191 stack_after=[pytuple], 1192 proto=1, 1193 doc="Push an empty tuple."), 1194 1195 I(name='TUPLE', 1196 code='t', 1197 arg=None, 1198 stack_before=[markobject, stackslice], 1199 stack_after=[pytuple], 1200 proto=0, 1201 doc="""Build a tuple out of the topmost stack slice, after markobject. 1202 1203 All the stack entries following the topmost markobject are placed into 1204 a single Python tuple, which single tuple object replaces all of the 1205 stack from the topmost markobject onward. For example, 1206 1207 Stack before: ... markobject 1 2 3 'abc' 1208 Stack after: ... (1, 2, 3, 'abc') 1209 """), 1210 1211 I(name='TUPLE1', 1212 code='\x85', 1213 arg=None, 1214 stack_before=[anyobject], 1215 stack_after=[pytuple], 1216 proto=2, 1217 doc="""Build a one-tuple out of the topmost item on the stack. 1218 1219 This code pops one value off the stack and pushes a tuple of 1220 length 1 whose one item is that value back onto it. In other 1221 words: 1222 1223 stack[-1] = tuple(stack[-1:]) 1224 """), 1225 1226 I(name='TUPLE2', 1227 code='\x86', 1228 arg=None, 1229 stack_before=[anyobject, anyobject], 1230 stack_after=[pytuple], 1231 proto=2, 1232 doc="""Build a two-tuple out of the top two items on the stack. 1233 1234 This code pops two values off the stack and pushes a tuple of 1235 length 2 whose items are those values back onto it. In other 1236 words: 1237 1238 stack[-2:] = [tuple(stack[-2:])] 1239 """), 1240 1241 I(name='TUPLE3', 1242 code='\x87', 1243 arg=None, 1244 stack_before=[anyobject, anyobject, anyobject], 1245 stack_after=[pytuple], 1246 proto=2, 1247 doc="""Build a three-tuple out of the top three items on the stack. 1248 1249 This code pops three values off the stack and pushes a tuple of 1250 length 3 whose items are those values back onto it. In other 1251 words: 1252 1253 stack[-3:] = [tuple(stack[-3:])] 1254 """), 1255 1256 # Ways to build dicts. 1257 1258 I(name='EMPTY_DICT', 1259 code='}', 1260 arg=None, 1261 stack_before=[], 1262 stack_after=[pydict], 1263 proto=1, 1264 doc="Push an empty dict."), 1265 1266 I(name='DICT', 1267 code='d', 1268 arg=None, 1269 stack_before=[markobject, stackslice], 1270 stack_after=[pydict], 1271 proto=0, 1272 doc="""Build a dict out of the topmost stack slice, after markobject. 1273 1274 All the stack entries following the topmost markobject are placed into 1275 a single Python dict, which single dict object replaces all of the 1276 stack from the topmost markobject onward. The stack slice alternates 1277 key, value, key, value, .... For example, 1278 1279 Stack before: ... markobject 1 2 3 'abc' 1280 Stack after: ... {1: 2, 3: 'abc'} 1281 """), 1282 1283 I(name='SETITEM', 1284 code='s', 1285 arg=None, 1286 stack_before=[pydict, anyobject, anyobject], 1287 stack_after=[pydict], 1288 proto=0, 1289 doc="""Add a key+value pair to an existing dict. 1290 1291 Stack before: ... pydict key value 1292 Stack after: ... pydict 1293 1294 where pydict has been modified via pydict[key] = value. 1295 """), 1296 1297 I(name='SETITEMS', 1298 code='u', 1299 arg=None, 1300 stack_before=[pydict, markobject, stackslice], 1301 stack_after=[pydict], 1302 proto=1, 1303 doc="""Add an arbitrary number of key+value pairs to an existing dict. 1304 1305 The slice of the stack following the topmost markobject is taken as 1306 an alternating sequence of keys and values, added to the dict 1307 immediately under the topmost markobject. Everything at and after the 1308 topmost markobject is popped, leaving the mutated dict at the top 1309 of the stack. 1310 1311 Stack before: ... pydict markobject key_1 value_1 ... key_n value_n 1312 Stack after: ... pydict 1313 1314 where pydict has been modified via pydict[key_i] = value_i for i in 1315 1, 2, ..., n, and in that order. 1316 """), 1317 1318 # Stack manipulation. 1319 1320 I(name='POP', 1321 code='0', 1322 arg=None, 1323 stack_before=[anyobject], 1324 stack_after=[], 1325 proto=0, 1326 doc="Discard the top stack item, shrinking the stack by one item."), 1327 1328 I(name='DUP', 1329 code='2', 1330 arg=None, 1331 stack_before=[anyobject], 1332 stack_after=[anyobject, anyobject], 1333 proto=0, 1334 doc="Push the top stack item onto the stack again, duplicating it."), 1335 1336 I(name='MARK', 1337 code='(', 1338 arg=None, 1339 stack_before=[], 1340 stack_after=[markobject], 1341 proto=0, 1342 doc="""Push markobject onto the stack. 1343 1344 markobject is a unique object, used by other opcodes to identify a 1345 region of the stack containing a variable number of objects for them 1346 to work on. See markobject.doc for more detail. 1347 """), 1348 1349 I(name='POP_MARK', 1350 code='1', 1351 arg=None, 1352 stack_before=[markobject, stackslice], 1353 stack_after=[], 1354 proto=1, 1355 doc="""Pop all the stack objects at and above the topmost markobject. 1356 1357 When an opcode using a variable number of stack objects is done, 1358 POP_MARK is used to remove those objects, and to remove the markobject 1359 that delimited their starting position on the stack. 1360 """), 1361 1362 # Memo manipulation. There are really only two operations (get and put), 1363 # each in all-text, "short binary", and "long binary" flavors. 1364 1365 I(name='GET', 1366 code='g', 1367 arg=decimalnl_short, 1368 stack_before=[], 1369 stack_after=[anyobject], 1370 proto=0, 1371 doc="""Read an object from the memo and push it on the stack. 1372 1373 The index of the memo object to push is given by the newline-terminated 1374 decimal string following. BINGET and LONG_BINGET are space-optimized 1375 versions. 1376 """), 1377 1378 I(name='BINGET', 1379 code='h', 1380 arg=uint1, 1381 stack_before=[], 1382 stack_after=[anyobject], 1383 proto=1, 1384 doc="""Read an object from the memo and push it on the stack. 1385 1386 The index of the memo object to push is given by the 1-byte unsigned 1387 integer following. 1388 """), 1389 1390 I(name='LONG_BINGET', 1391 code='j', 1392 arg=int4, 1393 stack_before=[], 1394 stack_after=[anyobject], 1395 proto=1, 1396 doc="""Read an object from the memo and push it on the stack. 1397 1398 The index of the memo object to push is given by the 4-byte signed 1399 little-endian integer following. 1400 """), 1401 1402 I(name='PUT', 1403 code='p', 1404 arg=decimalnl_short, 1405 stack_before=[], 1406 stack_after=[], 1407 proto=0, 1408 doc="""Store the stack top into the memo. The stack is not popped. 1409 1410 The index of the memo location to write into is given by the newline- 1411 terminated decimal string following. BINPUT and LONG_BINPUT are 1412 space-optimized versions. 1413 """), 1414 1415 I(name='BINPUT', 1416 code='q', 1417 arg=uint1, 1418 stack_before=[], 1419 stack_after=[], 1420 proto=1, 1421 doc="""Store the stack top into the memo. The stack is not popped. 1422 1423 The index of the memo location to write into is given by the 1-byte 1424 unsigned integer following. 1425 """), 1426 1427 I(name='LONG_BINPUT', 1428 code='r', 1429 arg=int4, 1430 stack_before=[], 1431 stack_after=[], 1432 proto=1, 1433 doc="""Store the stack top into the memo. The stack is not popped. 1434 1435 The index of the memo location to write into is given by the 4-byte 1436 signed little-endian integer following. 1437 """), 1438 1439 # Access the extension registry (predefined objects). Akin to the GET 1440 # family. 1441 1442 I(name='EXT1', 1443 code='\x82', 1444 arg=uint1, 1445 stack_before=[], 1446 stack_after=[anyobject], 1447 proto=2, 1448 doc="""Extension code. 1449 1450 This code and the similar EXT2 and EXT4 allow using a registry 1451 of popular objects that are pickled by name, typically classes. 1452 It is envisioned that through a global negotiation and 1453 registration process, third parties can set up a mapping between 1454 ints and object names. 1455 1456 In order to guarantee pickle interchangeability, the extension 1457 code registry ought to be global, although a range of codes may 1458 be reserved for private use. 1459 1460 EXT1 has a 1-byte integer argument. This is used to index into the 1461 extension registry, and the object at that index is pushed on the stack. 1462 """), 1463 1464 I(name='EXT2', 1465 code='\x83', 1466 arg=uint2, 1467 stack_before=[], 1468 stack_after=[anyobject], 1469 proto=2, 1470 doc="""Extension code. 1471 1472 See EXT1. EXT2 has a two-byte integer argument. 1473 """), 1474 1475 I(name='EXT4', 1476 code='\x84', 1477 arg=int4, 1478 stack_before=[], 1479 stack_after=[anyobject], 1480 proto=2, 1481 doc="""Extension code. 1482 1483 See EXT1. EXT4 has a four-byte integer argument. 1484 """), 1485 1486 # Push a class object, or module function, on the stack, via its module 1487 # and name. 1488 1489 I(name='GLOBAL', 1490 code='c', 1491 arg=stringnl_noescape_pair, 1492 stack_before=[], 1493 stack_after=[anyobject], 1494 proto=0, 1495 doc="""Push a global object (module.attr) on the stack. 1496 1497 Two newline-terminated strings follow the GLOBAL opcode. The first is 1498 taken as a module name, and the second as a class name. The class 1499 object module.class is pushed on the stack. More accurately, the 1500 object returned by self.find_class(module, class) is pushed on the 1501 stack, so unpickling subclasses can override this form of lookup. 1502 """), 1503 1504 # Ways to build objects of classes pickle doesn't know about directly 1505 # (user-defined classes). I despair of documenting this accurately 1506 # and comprehensibly -- you really have to read the pickle code to 1507 # find all the special cases. 1508 1509 I(name='REDUCE', 1510 code='R', 1511 arg=None, 1512 stack_before=[anyobject, anyobject], 1513 stack_after=[anyobject], 1514 proto=0, 1515 doc="""Push an object built from a callable and an argument tuple. 1516 1517 The opcode is named to remind of the __reduce__() method. 1518 1519 Stack before: ... callable pytuple 1520 Stack after: ... callable(*pytuple) 1521 1522 The callable and the argument tuple are the first two items returned 1523 by a __reduce__ method. Applying the callable to the argtuple is 1524 supposed to reproduce the original object, or at least get it started. 1525 If the __reduce__ method returns a 3-tuple, the last component is an 1526 argument to be passed to the object's __setstate__, and then the REDUCE 1527 opcode is followed by code to create setstate's argument, and then a 1528 BUILD opcode to apply __setstate__ to that argument. 1529 1530 If type(callable) is not ClassType, REDUCE complains unless the 1531 callable has been registered with the copy_reg module's 1532 safe_constructors dict, or the callable has a magic 1533 '__safe_for_unpickling__' attribute with a true value. I'm not sure 1534 why it does this, but I've sure seen this complaint often enough when 1535 I didn't want to <wink>. 1536 """), 1537 1538 I(name='BUILD', 1539 code='b', 1540 arg=None, 1541 stack_before=[anyobject, anyobject], 1542 stack_after=[anyobject], 1543 proto=0, 1544 doc="""Finish building an object, via __setstate__ or dict update. 1545 1546 Stack before: ... anyobject argument 1547 Stack after: ... anyobject 1548 1549 where anyobject may have been mutated, as follows: 1550 1551 If the object has a __setstate__ method, 1552 1553 anyobject.__setstate__(argument) 1554 1555 is called. 1556 1557 Else the argument must be a dict, the object must have a __dict__, and 1558 the object is updated via 1559 1560 anyobject.__dict__.update(argument) 1561 1562 This may raise RuntimeError in restricted execution mode (which 1563 disallows access to __dict__ directly); in that case, the object 1564 is updated instead via 1565 1566 for k, v in argument.items(): 1567 anyobject[k] = v 1568 """), 1569 1570 I(name='INST', 1571 code='i', 1572 arg=stringnl_noescape_pair, 1573 stack_before=[markobject, stackslice], 1574 stack_after=[anyobject], 1575 proto=0, 1576 doc="""Build a class instance. 1577 1578 This is the protocol 0 version of protocol 1's OBJ opcode. 1579 INST is followed by two newline-terminated strings, giving a 1580 module and class name, just as for the GLOBAL opcode (and see 1581 GLOBAL for more details about that). self.find_class(module, name) 1582 is used to get a class object. 1583 1584 In addition, all the objects on the stack following the topmost 1585 markobject are gathered into a tuple and popped (along with the 1586 topmost markobject), just as for the TUPLE opcode. 1587 1588 Now it gets complicated. If all of these are true: 1589 1590 + The argtuple is empty (markobject was at the top of the stack 1591 at the start). 1592 1593 + It's an old-style class object (the type of the class object is 1594 ClassType). 1595 1596 + The class object does not have a __getinitargs__ attribute. 1597 1598 then we want to create an old-style class instance without invoking 1599 its __init__() method (pickle has waffled on this over the years; not 1600 calling __init__() is current wisdom). In this case, an instance of 1601 an old-style dummy class is created, and then we try to rebind its 1602 __class__ attribute to the desired class object. If this succeeds, 1603 the new instance object is pushed on the stack, and we're done. In 1604 restricted execution mode it can fail (assignment to __class__ is 1605 disallowed), and I'm not really sure what happens then -- it looks 1606 like the code ends up calling the class object's __init__ anyway, 1607 via falling into the next case. 1608 1609 Else (the argtuple is not empty, it's not an old-style class object, 1610 or the class object does have a __getinitargs__ attribute), the code 1611 first insists that the class object have a __safe_for_unpickling__ 1612 attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, 1613 it doesn't matter whether this attribute has a true or false value, it 1614 only matters whether it exists (XXX this is a bug; cPickle 1615 requires the attribute to be true). If __safe_for_unpickling__ 1616 doesn't exist, UnpicklingError is raised. 1617 1618 Else (the class object does have a __safe_for_unpickling__ attr), 1619 the class object obtained from INST's arguments is applied to the 1620 argtuple obtained from the stack, and the resulting instance object 1621 is pushed on the stack. 1622 1623 NOTE: checks for __safe_for_unpickling__ went away in Python 2.3. 1624 """), 1625 1626 I(name='OBJ', 1627 code='o', 1628 arg=None, 1629 stack_before=[markobject, anyobject, stackslice], 1630 stack_after=[anyobject], 1631 proto=1, 1632 doc="""Build a class instance. 1633 1634 This is the protocol 1 version of protocol 0's INST opcode, and is 1635 very much like it. The major difference is that the class object 1636 is taken off the stack, allowing it to be retrieved from the memo 1637 repeatedly if several instances of the same class are created. This 1638 can be much more efficient (in both time and space) than repeatedly 1639 embedding the module and class names in INST opcodes. 1640 1641 Unlike INST, OBJ takes no arguments from the opcode stream. Instead 1642 the class object is taken off the stack, immediately above the 1643 topmost markobject: 1644 1645 Stack before: ... markobject classobject stackslice 1646 Stack after: ... new_instance_object 1647 1648 As for INST, the remainder of the stack above the markobject is 1649 gathered into an argument tuple, and then the logic seems identical, 1650 except that no __safe_for_unpickling__ check is done (XXX this is 1651 a bug; cPickle does test __safe_for_unpickling__). See INST for 1652 the gory details. 1653 1654 NOTE: In Python 2.3, INST and OBJ are identical except for how they 1655 get the class object. That was always the intent; the implementations 1656 had diverged for accidental reasons. 1657 """), 1658 1659 I(name='NEWOBJ', 1660 code='\x81', 1661 arg=None, 1662 stack_before=[anyobject, anyobject], 1663 stack_after=[anyobject], 1664 proto=2, 1665 doc="""Build an object instance. 1666 1667 The stack before should be thought of as containing a class 1668 object followed by an argument tuple (the tuple being the stack 1669 top). Call these cls and args. They are popped off the stack, 1670 and the value returned by cls.__new__(cls, *args) is pushed back 1671 onto the stack. 1672 """), 1673 1674 # Machine control. 1675 1676 I(name='PROTO', 1677 code='\x80', 1678 arg=uint1, 1679 stack_before=[], 1680 stack_after=[], 1681 proto=2, 1682 doc="""Protocol version indicator. 1683 1684 For protocol 2 and above, a pickle must start with this opcode. 1685 The argument is the protocol version, an int in range(2, 256). 1686 """), 1687 1688 I(name='STOP', 1689 code='.', 1690 arg=None, 1691 stack_before=[anyobject], 1692 stack_after=[], 1693 proto=0, 1694 doc="""Stop the unpickling machine. 1695 1696 Every pickle ends with this opcode. The object at the top of the stack 1697 is popped, and that's the result of unpickling. The stack should be 1698 empty then. 1699 """), 1700 1701 # Ways to deal with persistent IDs. 1702 1703 I(name='PERSID', 1704 code='P', 1705 arg=stringnl_noescape, 1706 stack_before=[], 1707 stack_after=[anyobject], 1708 proto=0, 1709 doc="""Push an object identified by a persistent ID. 1710 1711 The pickle module doesn't define what a persistent ID means. PERSID's 1712 argument is a newline-terminated str-style (no embedded escapes, no 1713 bracketing quote characters) string, which *is* "the persistent ID". 1714 The unpickler passes this string to self.persistent_load(). Whatever 1715 object that returns is pushed on the stack. There is no implementation 1716 of persistent_load() in Python's unpickler: it must be supplied by an 1717 unpickler subclass. 1718 """), 1719 1720 I(name='BINPERSID', 1721 code='Q', 1722 arg=None, 1723 stack_before=[anyobject], 1724 stack_after=[anyobject], 1725 proto=1, 1726 doc="""Push an object identified by a persistent ID. 1727 1728 Like PERSID, except the persistent ID is popped off the stack (instead 1729 of being a string embedded in the opcode bytestream). The persistent 1730 ID is passed to self.persistent_load(), and whatever object that 1731 returns is pushed on the stack. See PERSID for more detail. 1732 """), 1733] 1734del I 1735 1736# Verify uniqueness of .name and .code members. 1737name2i = {} 1738code2i = {} 1739 1740for i, d in enumerate(opcodes): 1741 if d.name in name2i: 1742 raise ValueError("repeated name %r at indices %d and %d" % 1743 (d.name, name2i[d.name], i)) 1744 if d.code in code2i: 1745 raise ValueError("repeated code %r at indices %d and %d" % 1746 (d.code, code2i[d.code], i)) 1747 1748 name2i[d.name] = i 1749 code2i[d.code] = i 1750 1751del name2i, code2i, i, d 1752 1753############################################################################## 1754# Build a code2op dict, mapping opcode characters to OpcodeInfo records. 1755# Also ensure we've got the same stuff as pickle.py, although the 1756# introspection here is dicey. 1757 1758code2op = {} 1759for d in opcodes: 1760 code2op[d.code] = d 1761del d 1762 1763def assure_pickle_consistency(verbose=False): 1764 import pickle, re 1765 1766 copy = code2op.copy() 1767 for name in pickle.__all__: 1768 if not re.match("[A-Z][A-Z0-9_]+$", name): 1769 if verbose: 1770 print "skipping %r: it doesn't look like an opcode name" % name 1771 continue 1772 picklecode = getattr(pickle, name) 1773 if not isinstance(picklecode, str) or len(picklecode) != 1: 1774 if verbose: 1775 print ("skipping %r: value %r doesn't look like a pickle " 1776 "code" % (name, picklecode)) 1777 continue 1778 if picklecode in copy: 1779 if verbose: 1780 print "checking name %r w/ code %r for consistency" % ( 1781 name, picklecode) 1782 d = copy[picklecode] 1783 if d.name != name: 1784 raise ValueError("for pickle code %r, pickle.py uses name %r " 1785 "but we're using name %r" % (picklecode, 1786 name, 1787 d.name)) 1788 # Forget this one. Any left over in copy at the end are a problem 1789 # of a different kind. 1790 del copy[picklecode] 1791 else: 1792 raise ValueError("pickle.py appears to have a pickle opcode with " 1793 "name %r and code %r, but we don't" % 1794 (name, picklecode)) 1795 if copy: 1796 msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"] 1797 for code, d in copy.items(): 1798 msg.append(" name %r with code %r" % (d.name, code)) 1799 raise ValueError("\n".join(msg)) 1800 1801assure_pickle_consistency() 1802del assure_pickle_consistency 1803 1804############################################################################## 1805# A pickle opcode generator. 1806 1807def genops(pickle): 1808 """Generate all the opcodes in a pickle. 1809 1810 'pickle' is a file-like object, or string, containing the pickle. 1811 1812 Each opcode in the pickle is generated, from the current pickle position, 1813 stopping after a STOP opcode is delivered. A triple is generated for 1814 each opcode: 1815 1816 opcode, arg, pos 1817 1818 opcode is an OpcodeInfo record, describing the current opcode. 1819 1820 If the opcode has an argument embedded in the pickle, arg is its decoded 1821 value, as a Python object. If the opcode doesn't have an argument, arg 1822 is None. 1823 1824 If the pickle has a tell() method, pos was the value of pickle.tell() 1825 before reading the current opcode. If the pickle is a string object, 1826 it's wrapped in a StringIO object, and the latter's tell() result is 1827 used. Else (the pickle doesn't have a tell(), and it's not obvious how 1828 to query its current position) pos is None. 1829 """ 1830 1831 import cStringIO as StringIO 1832 1833 if isinstance(pickle, str): 1834 pickle = StringIO.StringIO(pickle) 1835 1836 if hasattr(pickle, "tell"): 1837 getpos = pickle.tell 1838 else: 1839 getpos = lambda: None 1840 1841 while True: 1842 pos = getpos() 1843 code = pickle.read(1) 1844 opcode = code2op.get(code) 1845 if opcode is None: 1846 if code == "": 1847 raise ValueError("pickle exhausted before seeing STOP") 1848 else: 1849 raise ValueError("at position %s, opcode %r unknown" % ( 1850 pos is None and "<unknown>" or pos, 1851 code)) 1852 if opcode.arg is None: 1853 arg = None 1854 else: 1855 arg = opcode.arg.reader(pickle) 1856 yield opcode, arg, pos 1857 if code == '.': 1858 assert opcode.name == 'STOP' 1859 break 1860 1861############################################################################## 1862# A pickle optimizer. 1863 1864def optimize(p): 1865 'Optimize a pickle string by removing unused PUT opcodes' 1866 gets = set() # set of args used by a GET opcode 1867 puts = [] # (arg, startpos, stoppos) for the PUT opcodes 1868 prevpos = None # set to pos if previous opcode was a PUT 1869 for opcode, arg, pos in genops(p): 1870 if prevpos is not None: 1871 puts.append((prevarg, prevpos, pos)) 1872 prevpos = None 1873 if 'PUT' in opcode.name: 1874 prevarg, prevpos = arg, pos 1875 elif 'GET' in opcode.name: 1876 gets.add(arg) 1877 1878 # Copy the pickle string except for PUTS without a corresponding GET 1879 s = [] 1880 i = 0 1881 for arg, start, stop in puts: 1882 j = stop if (arg in gets) else start 1883 s.append(p[i:j]) 1884 i = stop 1885 s.append(p[i:]) 1886 return ''.join(s) 1887 1888############################################################################## 1889# A symbolic pickle disassembler. 1890 1891def dis(pickle, out=None, memo=None, indentlevel=4): 1892 """Produce a symbolic disassembly of a pickle. 1893 1894 'pickle' is a file-like object, or string, containing a (at least one) 1895 pickle. The pickle is disassembled from the current position, through 1896 the first STOP opcode encountered. 1897 1898 Optional arg 'out' is a file-like object to which the disassembly is 1899 printed. It defaults to sys.stdout. 1900 1901 Optional arg 'memo' is a Python dict, used as the pickle's memo. It 1902 may be mutated by dis(), if the pickle contains PUT or BINPUT opcodes. 1903 Passing the same memo object to another dis() call then allows disassembly 1904 to proceed across multiple pickles that were all created by the same 1905 pickler with the same memo. Ordinarily you don't need to worry about this. 1906 1907 Optional arg indentlevel is the number of blanks by which to indent 1908 a new MARK level. It defaults to 4. 1909 1910 In addition to printing the disassembly, some sanity checks are made: 1911 1912 + All embedded opcode arguments "make sense". 1913 1914 + Explicit and implicit pop operations have enough items on the stack. 1915 1916 + When an opcode implicitly refers to a markobject, a markobject is 1917 actually on the stack. 1918 1919 + A memo entry isn't referenced before it's defined. 1920 1921 + The markobject isn't stored in the memo. 1922 1923 + A memo entry isn't redefined. 1924 """ 1925 1926 # Most of the hair here is for sanity checks, but most of it is needed 1927 # anyway to detect when a protocol 0 POP takes a MARK off the stack 1928 # (which in turn is needed to indent MARK blocks correctly). 1929 1930 stack = [] # crude emulation of unpickler stack 1931 if memo is None: 1932 memo = {} # crude emulation of unpicker memo 1933 maxproto = -1 # max protocol number seen 1934 markstack = [] # bytecode positions of MARK opcodes 1935 indentchunk = ' ' * indentlevel 1936 errormsg = None 1937 for opcode, arg, pos in genops(pickle): 1938 if pos is not None: 1939 print >> out, "%5d:" % pos, 1940 1941 line = "%-4s %s%s" % (repr(opcode.code)[1:-1], 1942 indentchunk * len(markstack), 1943 opcode.name) 1944 1945 maxproto = max(maxproto, opcode.proto) 1946 before = opcode.stack_before # don't mutate 1947 after = opcode.stack_after # don't mutate 1948 numtopop = len(before) 1949 1950 # See whether a MARK should be popped. 1951 markmsg = None 1952 if markobject in before or (opcode.name == "POP" and 1953 stack and 1954 stack[-1] is markobject): 1955 assert markobject not in after 1956 if __debug__: 1957 if markobject in before: 1958 assert before[-1] is stackslice 1959 if markstack: 1960 markpos = markstack.pop() 1961 if markpos is None: 1962 markmsg = "(MARK at unknown opcode offset)" 1963 else: 1964 markmsg = "(MARK at %d)" % markpos 1965 # Pop everything at and after the topmost markobject. 1966 while stack[-1] is not markobject: 1967 stack.pop() 1968 stack.pop() 1969 # Stop later code from popping too much. 1970 try: 1971 numtopop = before.index(markobject) 1972 except ValueError: 1973 assert opcode.name == "POP" 1974 numtopop = 0 1975 else: 1976 errormsg = markmsg = "no MARK exists on stack" 1977 1978 # Check for correct memo usage. 1979 if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"): 1980 assert arg is not None 1981 if arg in memo: 1982 errormsg = "memo key %r already defined" % arg 1983 elif not stack: 1984 errormsg = "stack is empty -- can't store into memo" 1985 elif stack[-1] is markobject: 1986 errormsg = "can't store markobject in the memo" 1987 else: 1988 memo[arg] = stack[-1] 1989 1990 elif opcode.name in ("GET", "BINGET", "LONG_BINGET"): 1991 if arg in memo: 1992 assert len(after) == 1 1993 after = [memo[arg]] # for better stack emulation 1994 else: 1995 errormsg = "memo key %r has never been stored into" % arg 1996 1997 if arg is not None or markmsg: 1998 # make a mild effort to align arguments 1999 line += ' ' * (10 - len(opcode.name)) 2000 if arg is not None: 2001 line += ' ' + repr(arg) 2002 if markmsg: 2003 line += ' ' + markmsg 2004 print >> out, line 2005 2006 if errormsg: 2007 # Note that we delayed complaining until the offending opcode 2008 # was printed. 2009 raise ValueError(errormsg) 2010 2011 # Emulate the stack effects. 2012 if len(stack) < numtopop: 2013 raise ValueError("tries to pop %d items from stack with " 2014 "only %d items" % (numtopop, len(stack))) 2015 if numtopop: 2016 del stack[-numtopop:] 2017 if markobject in after: 2018 assert markobject not in before 2019 markstack.append(pos) 2020 2021 stack.extend(after) 2022 2023 print >> out, "highest protocol among opcodes =", maxproto 2024 if stack: 2025 raise ValueError("stack not empty after STOP: %r" % stack) 2026 2027# For use in the doctest, simply as an example of a class to pickle. 2028class _Example: 2029 def __init__(self, value): 2030 self.value = value 2031 2032_dis_test = r""" 2033>>> import pickle 2034>>> x = [1, 2, (3, 4), {'abc': u"def"}] 2035>>> pkl = pickle.dumps(x, 0) 2036>>> dis(pkl) 2037 0: ( MARK 2038 1: l LIST (MARK at 0) 2039 2: p PUT 0 2040 5: I INT 1 2041 8: a APPEND 2042 9: I INT 2 2043 12: a APPEND 2044 13: ( MARK 2045 14: I INT 3 2046 17: I INT 4 2047 20: t TUPLE (MARK at 13) 2048 21: p PUT 1 2049 24: a APPEND 2050 25: ( MARK 2051 26: d DICT (MARK at 25) 2052 27: p PUT 2 2053 30: S STRING 'abc' 2054 37: p PUT 3 2055 40: V UNICODE u'def' 2056 45: p PUT 4 2057 48: s SETITEM 2058 49: a APPEND 2059 50: . STOP 2060highest protocol among opcodes = 0 2061 2062Try again with a "binary" pickle. 2063 2064>>> pkl = pickle.dumps(x, 1) 2065>>> dis(pkl) 2066 0: ] EMPTY_LIST 2067 1: q BINPUT 0 2068 3: ( MARK 2069 4: K BININT1 1 2070 6: K BININT1 2 2071 8: ( MARK 2072 9: K BININT1 3 2073 11: K BININT1 4 2074 13: t TUPLE (MARK at 8) 2075 14: q BINPUT 1 2076 16: } EMPTY_DICT 2077 17: q BINPUT 2 2078 19: U SHORT_BINSTRING 'abc' 2079 24: q BINPUT 3 2080 26: X BINUNICODE u'def' 2081 34: q BINPUT 4 2082 36: s SETITEM 2083 37: e APPENDS (MARK at 3) 2084 38: . STOP 2085highest protocol among opcodes = 1 2086 2087Exercise the INST/OBJ/BUILD family. 2088 2089>>> import pickletools 2090>>> dis(pickle.dumps(pickletools.dis, 0)) 2091 0: c GLOBAL 'pickletools dis' 2092 17: p PUT 0 2093 20: . STOP 2094highest protocol among opcodes = 0 2095 2096>>> from pickletools import _Example 2097>>> x = [_Example(42)] * 2 2098>>> dis(pickle.dumps(x, 0)) 2099 0: ( MARK 2100 1: l LIST (MARK at 0) 2101 2: p PUT 0 2102 5: ( MARK 2103 6: i INST 'pickletools _Example' (MARK at 5) 2104 28: p PUT 1 2105 31: ( MARK 2106 32: d DICT (MARK at 31) 2107 33: p PUT 2 2108 36: S STRING 'value' 2109 45: p PUT 3 2110 48: I INT 42 2111 52: s SETITEM 2112 53: b BUILD 2113 54: a APPEND 2114 55: g GET 1 2115 58: a APPEND 2116 59: . STOP 2117highest protocol among opcodes = 0 2118 2119>>> dis(pickle.dumps(x, 1)) 2120 0: ] EMPTY_LIST 2121 1: q BINPUT 0 2122 3: ( MARK 2123 4: ( MARK 2124 5: c GLOBAL 'pickletools _Example' 2125 27: q BINPUT 1 2126 29: o OBJ (MARK at 4) 2127 30: q BINPUT 2 2128 32: } EMPTY_DICT 2129 33: q BINPUT 3 2130 35: U SHORT_BINSTRING 'value' 2131 42: q BINPUT 4 2132 44: K BININT1 42 2133 46: s SETITEM 2134 47: b BUILD 2135 48: h BINGET 2 2136 50: e APPENDS (MARK at 3) 2137 51: . STOP 2138highest protocol among opcodes = 1 2139 2140Try "the canonical" recursive-object test. 2141 2142>>> L = [] 2143>>> T = L, 2144>>> L.append(T) 2145>>> L[0] is T 2146True 2147>>> T[0] is L 2148True 2149>>> L[0][0] is L 2150True 2151>>> T[0][0] is T 2152True 2153>>> dis(pickle.dumps(L, 0)) 2154 0: ( MARK 2155 1: l LIST (MARK at 0) 2156 2: p PUT 0 2157 5: ( MARK 2158 6: g GET 0 2159 9: t TUPLE (MARK at 5) 2160 10: p PUT 1 2161 13: a APPEND 2162 14: . STOP 2163highest protocol among opcodes = 0 2164 2165>>> dis(pickle.dumps(L, 1)) 2166 0: ] EMPTY_LIST 2167 1: q BINPUT 0 2168 3: ( MARK 2169 4: h BINGET 0 2170 6: t TUPLE (MARK at 3) 2171 7: q BINPUT 1 2172 9: a APPEND 2173 10: . STOP 2174highest protocol among opcodes = 1 2175 2176Note that, in the protocol 0 pickle of the recursive tuple, the disassembler 2177has to emulate the stack in order to realize that the POP opcode at 16 gets 2178rid of the MARK at 0. 2179 2180>>> dis(pickle.dumps(T, 0)) 2181 0: ( MARK 2182 1: ( MARK 2183 2: l LIST (MARK at 1) 2184 3: p PUT 0 2185 6: ( MARK 2186 7: g GET 0 2187 10: t TUPLE (MARK at 6) 2188 11: p PUT 1 2189 14: a APPEND 2190 15: 0 POP 2191 16: 0 POP (MARK at 0) 2192 17: g GET 1 2193 20: . STOP 2194highest protocol among opcodes = 0 2195 2196>>> dis(pickle.dumps(T, 1)) 2197 0: ( MARK 2198 1: ] EMPTY_LIST 2199 2: q BINPUT 0 2200 4: ( MARK 2201 5: h BINGET 0 2202 7: t TUPLE (MARK at 4) 2203 8: q BINPUT 1 2204 10: a APPEND 2205 11: 1 POP_MARK (MARK at 0) 2206 12: h BINGET 1 2207 14: . STOP 2208highest protocol among opcodes = 1 2209 2210Try protocol 2. 2211 2212>>> dis(pickle.dumps(L, 2)) 2213 0: \x80 PROTO 2 2214 2: ] EMPTY_LIST 2215 3: q BINPUT 0 2216 5: h BINGET 0 2217 7: \x85 TUPLE1 2218 8: q BINPUT 1 2219 10: a APPEND 2220 11: . STOP 2221highest protocol among opcodes = 2 2222 2223>>> dis(pickle.dumps(T, 2)) 2224 0: \x80 PROTO 2 2225 2: ] EMPTY_LIST 2226 3: q BINPUT 0 2227 5: h BINGET 0 2228 7: \x85 TUPLE1 2229 8: q BINPUT 1 2230 10: a APPEND 2231 11: 0 POP 2232 12: h BINGET 1 2233 14: . STOP 2234highest protocol among opcodes = 2 2235""" 2236 2237_memo_test = r""" 2238>>> import pickle 2239>>> from StringIO import StringIO 2240>>> f = StringIO() 2241>>> p = pickle.Pickler(f, 2) 2242>>> x = [1, 2, 3] 2243>>> p.dump(x) 2244>>> p.dump(x) 2245>>> f.seek(0) 2246>>> memo = {} 2247>>> dis(f, memo=memo) 2248 0: \x80 PROTO 2 2249 2: ] EMPTY_LIST 2250 3: q BINPUT 0 2251 5: ( MARK 2252 6: K BININT1 1 2253 8: K BININT1 2 2254 10: K BININT1 3 2255 12: e APPENDS (MARK at 5) 2256 13: . STOP 2257highest protocol among opcodes = 2 2258>>> dis(f, memo=memo) 2259 14: \x80 PROTO 2 2260 16: h BINGET 0 2261 18: . STOP 2262highest protocol among opcodes = 2 2263""" 2264 2265__test__ = {'disassembler_test': _dis_test, 2266 'disassembler_memo_test': _memo_test, 2267 } 2268 2269def _test(): 2270 import doctest 2271 return doctest.testmod() 2272 2273if __name__ == "__main__": 2274 _test() 2275