...
 
Commits (9)
+ Version 2.18 (04.07.2017)
- PR #161 & #184: Update bundled PLY version to 3.10
- PR #158: Add support for the __int128 type.
- PR #169: Handle more tricky TYPEID in declarators.
- PR #178: Add columns to the coord of each node
+ Version 2.17 (29.10.2016)
- Again functionality identical to 2.15 and 2.16; the difference is that the
......
Copyright (c) 2008-2016, Eli Bendersky
pycparser -- A C parser in Python
Copyright (c) 2008-2017, Eli Bendersky
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
......
Metadata-Version: 1.1
Name: pycparser
Version: 2.17
Version: 2.18
Summary: C parser in Python
Home-page: https://github.com/eliben/pycparser
Author: Eli Bendersky
......
===============
pycparser v2.17
pycparser v2.18
===============
:Author: `Eli Bendersky <http://eli.thegreenplace.net>`_
......@@ -57,20 +57,20 @@ things up so that it parses code with a lot of GCC-isms successfully. See the
What grammar does pycparser follow?
-----------------------------------
**pycparser** very closely follows the C grammar provided in the end of the C99
standard document
**pycparser** very closely follows the C grammar provided in Annex A of the C99
standard (ISO/IEC 9899).
How is pycparser licensed?
--------------------------
BSD license. See the `LICENSE` file in the distribution.
`BSD license <https://github.com/eliben/pycparser/blob/master/LICENSE>`_.
Contact details
---------------
Drop me an email to eliben@gmail.com for any questions regarding **pycparser**.
For reporting problems with **pycparser** or submitting feature requests, the
best way is to open an `issue <https://github.com/eliben/pycparser/issues>`_.
For reporting problems with **pycparser** or submitting feature requests, please
open an `issue <https://github.com/eliben/pycparser/issues>`_, or submit a
pull request.
Installing
......@@ -85,7 +85,7 @@ Prerequisites
* **pycparser** has no external dependencies. The only non-stdlib library it
uses is PLY, which is bundled in ``pycparser/ply``. The current PLY version is
3.8, retrieved from `<http://www.dabeaz.com/ply/ply-3.8.tar.gz>`_
3.10, retrieved from `<http://www.dabeaz.com/ply/>`_
Installation process
--------------------
......@@ -111,6 +111,7 @@ Known problems
deleting the ``pycparser`` directory in your Python's ``site-packages``, or
wherever you installed it) and install again.
Using
=====
......@@ -119,10 +120,10 @@ Interaction with the C preprocessor
In order to be compilable, C code must be preprocessed by the C preprocessor -
``cpp``. ``cpp`` handles preprocessing directives like ``#include`` and
``#define``, removes comments, and does other minor tasks that prepare the C
``#define``, removes comments, and performs other minor tasks that prepare the C
code for compilation.
For all but the most trivial snippets of C code, **pycparser**, like a C
For all but the most trivial snippets of C code **pycparser**, like a C
compiler, must receive preprocessed C code in order to function correctly. If
you import the top-level ``parse_file`` function from the **pycparser** package,
it will interact with ``cpp`` for you, as long as it's in your PATH, or you
......@@ -136,13 +137,13 @@ and install a binary build of Clang for Windows `from this website
What about the standard C library headers?
------------------------------------------
C code almost always includes various header files from the standard C library,
like ``stdio.h``. While, with some effort, **pycparser** can be made to parse
the standard headers from any C compiler, it's much simpler to use the provided
"fake" standard includes in ``utils/fake_libc_include``. These are standard C
header files that contain only the bare necessities to allow valid parsing of
the files that use them. As a bonus, since they're minimal, it can significantly
improve the performance of parsing large C files.
C code almost always ``#include``\s various header files from the standard C
library, like ``stdio.h``. While (with some effort) **pycparser** can be made to
parse the standard headers from any C compiler, it's much simpler to use the
provided "fake" standard includes in ``utils/fake_libc_include``. These are
standard C header files that contain only the bare necessities to allow valid
parsing of the files that use them. As a bonus, since they're minimal, it can
significantly improve the performance of parsing large C files.
The key point to understand here is that **pycparser** doesn't really care about
the semantics of types. It only needs to know whether some token encountered in
......@@ -169,6 +170,7 @@ created by the parser, see ``pycparser/_c_ast.cfg``.
There's also a `FAQ available here <https://github.com/eliben/pycparser/wiki/FAQ>`_.
In any case, you can always drop me an `email <eliben@gmail.com>`_ for help.
Modifying
=========
......@@ -213,15 +215,17 @@ utils/fake_libc_include:
utils/internal/:
Internal utilities for my own use. You probably don't need them.
Contributors
============
Some people have contributed to **pycparser** by opening issues on bugs they've
found and/or submitting patches. The list of contributors is in the CONTRIBUTORS
file in the source distribution. Once **pycparser** moved to Github, I stopped
file in the source distribution. After **pycparser** moved to Github I stopped
updating this list because Github does a much better job at tracking
contributions.
CI Status
=========
......@@ -232,3 +236,8 @@ CI Status
:align: center
:target: https://travis-ci.org/eliben/pycparser
AppVeyor also helps run tests on Windows:
.. image:: https://ci.appveyor.com/api/projects/status/wrup68o5y8nuk1i9?svg=true
:align: center
:target: https://ci.appveyor.com/project/eliben/pycparser/
# see git-dpm(1) from git-dpm package
4e0b47ad9bd463acfcb8736e027b5e005dcf9aaf
4e0b47ad9bd463acfcb8736e027b5e005dcf9aaf
73d0dd760e0d6d9bc5b250f56b8b8ca8311f7ba2
73d0dd760e0d6d9bc5b250f56b8b8ca8311f7ba2
pycparser_2.17.orig.tar.gz
afc1f70c6fce0e196dece88d6da3b8d5b2af6297
231163
debianTag="debian/%e%v"
patchedTag="patched/%e%v"
upstreamTag="upstream/%e%u"
pycparser (2.17-3) UNRELEASED; urgency=medium
pycparser (2.18-1) unstable; urgency=medium
[ Stefano Rivera ]
* New upstream release.
* Refresh patches.
* Update copyright.
* Bump Standards-Version to 4.1.3, no changes needed.
* Bump debhelper compat level to 11.
[ Ondřej Nový ]
* d/control: Set Vcs-* to salsa.debian.org
* d/copyright: Use https protocol in Format field
-- Ondřej Nový <onovy@debian.org> Tue, 13 Feb 2018 10:12:37 +0100
-- Stefano Rivera <stefanor@debian.org> Sun, 18 Feb 2018 23:53:54 -0800
pycparser (2.17-2) unstable; urgency=medium
......
......@@ -4,13 +4,13 @@ Priority: optional
Maintainer: Debian Python Modules Team <python-modules-team@lists.alioth.debian.org>
Uploaders: Stefano Rivera <stefanor@debian.org>
Build-Depends:
debhelper (>= 10),
debhelper (>= 11),
dh-python (>= 1.20140511),
python-all,
python-ply,
python3-all,
python3-ply
Standards-Version: 3.9.8
Standards-Version: 4.1.3
Homepage: https://github.com/eliben/pycparser
X-Python-Version: >= 2.6
X-Python3-Version: >= 3.2
......
......@@ -4,19 +4,15 @@ Upstream-Contact: Eli Bendersky <eliben@gmail.com>
Source: http://pypi.python.org/pypi/pycparser
Files: *
Copyright: 2008-2016, Eli Bendersky <eliben@gmail.com>
Copyright: 2008-2017, Eli Bendersky <eliben@gmail.com>
License: bsd-3-bendersky
Files: pycparser/ply/*
Copyright: 2001-2012 David M. Beazley (Dabeaz LLC) <dave@dabeaz.com>
Copyright: 2001-2017 David M. Beazley (Dabeaz LLC) <dave@dabeaz.com>
License: bsd-3-dabeaz
Files: examples/rewrite_ast.py
Copyright: 2014, Akira Hayakawa
License: bsd-3-bendersky
Files: debian/*
Copyright: 2012-2016, Stefano Rivera <stefanor@debian.org>
Copyright: 2012-2018, Stefano Rivera <stefanor@debian.org>
License: bsd-3-bendersky
License: bsd-3-bendersky
......
From 4e0b47ad9bd463acfcb8736e027b5e005dcf9aaf Mon Sep 17 00:00:00 2001
From: Stefano Rivera <stefanor@debian.org>
Date: Thu, 8 Oct 2015 10:23:38 -0700
Subject: Use Debian's ply instead of the bundled ply
We prefer to avoid bundled libraries in Debian, don't use the bundled ply.
We prefer to avoid bundled libraries in Debian, don't use the bundled
ply.
Last-Update: 2013-01-12
Forwarded: not-needed
Patch-Name: system-ply
---
pycparser/c_lexer.py | 4 ++--
pycparser/c_parser.py | 2 +-
......@@ -15,7 +14,7 @@ Patch-Name: system-ply
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/pycparser/c_lexer.py b/pycparser/c_lexer.py
index c7443b8..8399a17 100644
index d9941c1..2c05cbb 100644
--- a/pycparser/c_lexer.py
+++ b/pycparser/c_lexer.py
@@ -9,8 +9,8 @@
......@@ -30,7 +29,7 @@ index c7443b8..8399a17 100644
class CLexer(object):
diff --git a/pycparser/c_parser.py b/pycparser/c_parser.py
index dc13962..ed581d5 100644
index f84d6bc..b1ece10 100644
--- a/pycparser/c_parser.py
+++ b/pycparser/c_parser.py
@@ -8,7 +8,7 @@
......@@ -43,7 +42,7 @@ index dc13962..ed581d5 100644
from . import c_ast
from .c_lexer import CLexer
diff --git a/setup.py b/setup.py
index 07a62da..a6cb267 100644
index 2ceb7b8..c83f6c8 100644
--- a/setup.py
+++ b/setup.py
@@ -49,7 +49,7 @@ setup(
......
......@@ -4,7 +4,7 @@
# Example of using pycparser.c_generator, serving as a simplistic translator
# from C to AST and back to C.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#------------------------------------------------------------------------------
from __future__ import print_function
......
#------------------------------------------------------------------------------
# pycparser: c_json.py
#
# by Michael White (@mypalmike)
#
# This example includes functions to serialize and deserialize an ast
# to and from json format. Serializing involves walking the ast and converting
# each node from a python Node object into a python dict. Deserializing
# involves the opposite conversion, walking the tree formed by the
# dict and converting each dict into the specific Node object it represents.
# The dict itself is serialized and deserialized using the python json module.
#
# The dict representation is a fairly direct transformation of the object
# attributes. Each node in the dict gets one metadata field referring to the
# specific node class name, _nodetype. Each local attribute (i.e. not linking
# to child nodes) has a string value or array of string values. Each child
# attribute is either another dict or an array of dicts, exactly as in the
# Node object representation. The "coord" attribute, representing the
# node's location within the source code, is serialized/deserialized from
# a Coord object into a string of the format "filename:line[:column]".
#
# Example TypeDecl node, with IdentifierType child node, represented as a dict:
# "type": {
# "_nodetype": "TypeDecl",
# "coord": "c_files/funky.c:8",
# "declname": "o",
# "quals": [],
# "type": {
# "_nodetype": "IdentifierType",
# "coord": "c_files/funky.c:8",
# "names": [
# "char"
# ]
# }
# }
#------------------------------------------------------------------------------
from __future__ import print_function
import json
import sys
import re
# This is not required if you've installed pycparser into
# your site-packages/ with setup.py
#
sys.path.extend(['.', '..'])
from pycparser import parse_file, c_ast
from pycparser.plyparser import Coord
RE_CHILD_ARRAY = re.compile(r'(.*)\[(.*)\]')
RE_INTERNAL_ATTR = re.compile('__.*__')
class CJsonError(Exception):
pass
def memodict(fn):
""" Fast memoization decorator for a function taking a single argument """
class memodict(dict):
def __missing__(self, key):
ret = self[key] = fn(key)
return ret
return memodict().__getitem__
@memodict
def child_attrs_of(klass):
"""
Given a Node class, get a set of child attrs.
Memoized to avoid highly repetitive string manipulation
"""
non_child_attrs = set(klass.attr_names)
all_attrs = set([i for i in klass.__slots__ if not RE_INTERNAL_ATTR.match(i)])
return all_attrs - non_child_attrs
def to_dict(node):
""" Recursively convert an ast into dict representation. """
klass = node.__class__
result = {}
# Metadata
result['_nodetype'] = klass.__name__
# Local node attributes
for attr in klass.attr_names:
result[attr] = getattr(node, attr)
# Coord object
if node.coord:
result['coord'] = str(node.coord)
else:
result['coord'] = None
# Child attributes
for child_name, child in node.children():
# Child strings are either simple (e.g. 'value') or arrays (e.g. 'block_items[1]')
match = RE_CHILD_ARRAY.match(child_name)
if match:
array_name, array_index = match.groups()
array_index = int(array_index)
# arrays come in order, so we verify and append.
result[array_name] = result.get(array_name, [])
if array_index != len(result[array_name]):
raise CJsonError('Internal ast error. Array {} out of order. '
'Expected index {}, got {}'.format(
array_name, len(result[array_name]), array_index))
result[array_name].append(to_dict(child))
else:
result[child_name] = to_dict(child)
# Any child attributes that were missing need "None" values in the json.
for child_attr in child_attrs_of(klass):
if child_attr not in result:
result[child_attr] = None
return result
def to_json(node, **kwargs):
""" Convert ast node to json string """
return json.dumps(to_dict(node), **kwargs)
def file_to_dict(filename):
""" Load C file into dict representation of ast """
ast = parse_file(filename, use_cpp=True)
return to_dict(ast)
def file_to_json(filename, **kwargs):
""" Load C file into json string representation of ast """
ast = parse_file(filename, use_cpp=True)
return to_json(ast, **kwargs)
def _parse_coord(coord_str):
""" Parse coord string (file:line[:column]) into Coord object. """
if coord_str is None:
return None
vals = coord_str.split(':')
vals.extend([None] * 3)
filename, line, column = vals[:3]
return Coord(filename, line, column)
def _convert_to_obj(value):
"""
Convert an object in the dict representation into an object.
Note: Mutually recursive with from_dict.
"""
value_type = type(value)
if value_type == dict:
return from_dict(value)
elif value_type == list:
return [_convert_to_obj(item) for item in value]
else:
# String
return value
def from_dict(node_dict):
""" Recursively build an ast from dict representation """
class_name = node_dict.pop('_nodetype')
klass = getattr(c_ast, class_name)
# Create a new dict containing the key-value pairs which we can pass
# to node constructors.
objs = {}
for key, value in node_dict.items():
if key == 'coord':
objs[key] = _parse_coord(value)
else:
objs[key] = _convert_to_obj(value)
# Use keyword parameters, which works thanks to beautifully consistent
# ast Node initializers.
return klass(**objs)
def from_json(ast_json):
""" Build an ast from json string representation """
return from_dict(json.loads(ast_json))
#------------------------------------------------------------------------------
if __name__ == "__main__":
if len(sys.argv) > 1:
# Some test code...
# Do trip from C -> ast -> dict -> ast -> json, then print.
ast_dict = file_to_dict(sys.argv[1])
ast = from_dict(ast_dict)
print(to_json(ast, sort_keys=True, indent=4))
else:
print("Please provide a filename as argument")
......@@ -6,19 +6,33 @@
#
# The AST generated by pycparser from the given declaration is traversed
# recursively to build the explanation. Note that the declaration must be a
# valid external declaration in C. All the types used in it must be defined with
# typedef, or parsing will fail. The definition can be arbitrary - pycparser
# doesn't really care what the type is defined to be, only that it's a type.
# valid external declaration in C. As shown below, typedef can be optionally
# expanded.
#
# For example:
#
# 'typedef int Node; const Node* (*ar)[10];'
# =>
# ar is a pointer to array[10] of pointer to const Node
# c_decl = 'typedef int Node; const Node* (*ar)[10];'
#
# Copyright (C) 2008-2015, Eli Bendersky
# explain_c_declaration(c_decl)
# => ar is a pointer to array[10] of pointer to const Node
#
# struct and typedef can be optionally expanded:
#
# explain_c_declaration(c_decl, expand_typedef=True)
# => ar is a pointer to array[10] of pointer to const int
#
# c_decl = 'struct P {int x; int y;} p;'
#
# explain_c_declaration(c_decl)
# => p is a struct P
#
# explain_c_declaration(c_decl, expand_struct=True)
# => p is a struct P containing {x is a int, y is a int}
#
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
import copy
import sys
# This is not required if you've installed pycparser into
......@@ -29,12 +43,15 @@ sys.path.extend(['.', '..'])
from pycparser import c_parser, c_ast
def explain_c_declaration(c_decl):
def explain_c_declaration(c_decl, expand_struct=False, expand_typedef=False):
""" Parses the declaration in c_decl and returns a text
explanation as a string.
The last external node of the string is used, to allow
earlier typedefs for used types.
The last external node of the string is used, to allow earlier typedefs
for used types.
expand_struct=True will spell out struct definitions recursively.
expand_typedef=True will expand typedef'd types.
"""
parser = c_parser.CParser()
......@@ -49,7 +66,14 @@ def explain_c_declaration(c_decl):
):
return "Not a valid declaration"
return _explain_decl_node(node.ext[-1])
try:
expanded = expand_struct_typedef(node.ext[-1], node,
expand_struct=expand_struct,
expand_typedef=expand_typedef)
except Exception as e:
return "Not a valid declaration: " + str(e)
return _explain_decl_node(expanded)
def _explain_decl_node(decl_node):
......@@ -95,6 +119,75 @@ def _explain_type(decl):
return ('function(%s) returning ' % (args) +
_explain_type(decl.type))
elif typ == c_ast.Struct:
decls = [_explain_decl_node(mem_decl) for mem_decl in decl.decls]
members = ', '.join(decls)
return ('struct%s ' % (' ' + decl.name if decl.name else '') +
('containing {%s}' % members if members else ''))
def expand_struct_typedef(cdecl, file_ast,
expand_struct=False,
expand_typedef=False):
"""Expand struct & typedef and return a new expanded node."""
decl_copy = copy.deepcopy(cdecl)
_expand_in_place(decl_copy, file_ast, expand_struct, expand_typedef)
return decl_copy
def _expand_in_place(decl, file_ast, expand_struct=False, expand_typedef=False):
"""Recursively expand struct & typedef in place, throw RuntimeError if
undeclared struct or typedef are used
"""
typ = type(decl)
if typ in (c_ast.Decl, c_ast.TypeDecl, c_ast.PtrDecl, c_ast.ArrayDecl):
decl.type = _expand_in_place(decl.type, file_ast, expand_struct,
expand_typedef)
elif typ == c_ast.Struct:
if not decl.decls:
struct = _find_struct(decl.name, file_ast)
if not struct:
raise RuntimeError('using undeclared struct %s' % decl.name)
decl.decls = struct.decls
for i, mem_decl in enumerate(decl.decls):
decl.decls[i] = _expand_in_place(mem_decl, file_ast, expand_struct,
expand_typedef)
if not expand_struct:
decl.decls = []
elif (typ == c_ast.IdentifierType and
decl.names[0] not in ('int', 'char')):
typedef = _find_typedef(decl.names[0], file_ast)
if not typedef:
raise RuntimeError('using undeclared type %s' % decl.names[0])
if expand_typedef:
return typedef.type
return decl
def _find_struct(name, file_ast):
"""Receives a struct name and return declared struct object in file_ast
"""
for node in file_ast.ext:
if (type(node) == c_ast.Decl and
type(node.type) == c_ast.Struct and
node.type.name == name):
return node.type
def _find_typedef(name, file_ast):
"""Receives a type name and return typedef object in file_ast
"""
for node in file_ast.ext:
if type(node) == c_ast.Typedef and node.name == name:
return node
if __name__ == "__main__":
if len(sys.argv) > 1:
......
#-----------------------------------------------------------------
# pycparser: dump_ast.py
#
# Basic example of parsing a file and dumping its parsed AST.
#
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
from __future__ import print_function
import argparse
import sys
# This is not required if you've installed pycparser into
# your site-packages/ with setup.py
sys.path.extend(['.', '..'])
from pycparser import c_parser, c_ast, parse_file
if __name__ == "__main__":
argparser = argparse.ArgumentParser('Dump AST')
argparser.add_argument('filename', help='name of file to parse')
args = argparser.parse_args()
ast = parse_file(args.filename, use_cpp=False)
ast.show()
......@@ -9,7 +9,7 @@
# information from the AST.
# It helps to have the pycparser/_c_ast.cfg file in front of you.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
from __future__ import print_function
......@@ -32,7 +32,7 @@ from pycparser import c_parser, c_ast
# to, so I've inserted the dummy typedef in the code to let the
# parser know Hash and Node are types. You don't need to do it
# when parsing real, correct C code.
#
text = r"""
typedef int Node, Hash;
......@@ -66,8 +66,8 @@ ast = parser.parse(text, filename='<none>')
# readable way. show() is the most useful tool in exploring ASTs
# created by pycparser. See the c_ast.py file for the options you
# can pass it.
#
#~ ast.show()
#ast.show(showcoord=True)
# OK, we've seen that the top node is FileAST. This is always the
# top node of the AST. Its children are "external declarations",
......@@ -79,48 +79,49 @@ ast = parser.parse(text, filename='<none>')
# ext[] holds the children of FileAST. Since the function
# definition is the third child, it's ext[2]. Uncomment the
# following line to show it:
#
#~ ast.ext[2].show()
#ast.ext[2].show()
# A FuncDef consists of a declaration, a list of parameter
# declarations (for K&R style function definitions), and a body.
# First, let's examine the declaration.
#
function_decl = ast.ext[2].decl
# function_decl, like any other declaration, is a Decl. Its type child
# is a FuncDecl, which has a return type and arguments stored in a
# ParamList node
#~ function_decl.type.show()
#~ function_decl.type.args.show()
#function_decl.type.show()
#function_decl.type.args.show()
# The following displays the name and type of each argument:
#
#~ for param_decl in function_decl.type.args.params:
#~ print('Arg name: %s' % param_decl.name)
#~ print('Type:')
#~ param_decl.type.show(offset=6)
#for param_decl in function_decl.type.args.params:
#print('Arg name: %s' % param_decl.name)
#print('Type:')
#param_decl.type.show(offset=6)
# The body is of FuncDef is a Compound, which is a placeholder for a block
# surrounded by {} (You should be reading _c_ast.cfg parallel to this
# explanation and seeing these things with your own eyes).
# Let's see the block's declarations:
#
function_body = ast.ext[2].body
# The following displays the declarations and statements in the function
# body
#
#~ for decl in function_body.block_items:
#~ decl.show()
#for decl in function_body.block_items:
#decl.show()
# We can see a single variable declaration, i, declared to be a simple type
# declaration of type 'unsigned int', followed by statements.
# block_items is a list, so the third element is the For statement:
#
for_stmt = function_body.block_items[2]
#~ for_stmt.show()
#for_stmt.show()
# As you can see in _c_ast.cfg, For's children are 'init, cond,
# next' for the respective parts of the 'for' loop specifier,
......@@ -128,26 +129,26 @@ for_stmt = function_body.block_items[2]
# a block.
#
# Let's dig deeper, to the while statement inside the for loop:
#
while_stmt = for_stmt.stmt.block_items[1]
#~ while_stmt.show()
#while_stmt.show()
# While is simpler, it only has a condition node and a stmt node.
# The condition:
#
while_cond = while_stmt.cond
#~ while_cond.show()
#while_cond.show()
# Note that it's a BinaryOp node - the basic constituent of
# expressions in our AST. BinaryOp is the expression tree, with
# left and right nodes as children. It also has the op attribute,
# which is just the string representation of the operator.
#
#~ print(while_cond.op)
#~ while_cond.left.show()
#~ while_cond.right.show()
#
#print(while_cond.op)
#while_cond.left.show()
#while_cond.right.show()
# That's it for the example. I hope you now see how easy it is to explore the
# AST created by pycparser. Although on the surface it is quite complex and has
# a lot of node types, this is the inherent complexity of the C language every
......@@ -156,6 +157,3 @@ while_cond = while_stmt.cond
# structure of AST nodes and write code that processes them.
# Specifically, see the cdecl.py example for a non-trivial demonstration of what
# you can do by recursively going through the AST.
#
#-----------------------------------------------------------------
# pycparser: func_defs.py
# pycparser: func_calls.py
#
# Using pycparser for printing out all the calls of some function
# in a C file.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
from __future__ import print_function
......
......@@ -7,7 +7,7 @@
# This is a simple example of traversing the AST generated by
# pycparser. Call it from the root directory of pycparser.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
from __future__ import print_function
......
#-----------------------------------------------------------------
# pycparser: func_write.py
# pycparser: rewrite_ast.py
#
# Tiny example of rewriting a AST node
#
# Copyright (C) 2014, Akira Hayakawa
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
from __future__ import print_function
......
#-----------------------------------------------------------------
# pycparser: serialize_ast.py
#
# Simple example of serializing AST
#
# Hart Chu [https://github.com/CtheSky]
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
from __future__ import print_function
import pickle
from pycparser import c_parser
text = r"""
void func(void)
{
x = 1;
}
"""
parser = c_parser.CParser()
ast = parser.parse(text)
# Since AST nodes use __slots__ for faster attribute access and
# space saving, it needs Pickle's protocol version >= 2.
# The default version is 3 for python 3.x and 1 for python 2.7.
# You can always select the highest available protocol with the -1 argument.
with open('ast', 'wb') as f:
pickle.dump(ast, f, protocol=-1)
# Deserialize.
with open('ast', 'rb') as f:
ast = pickle.load(f)
ast.show()
......@@ -5,7 +5,7 @@
# the 'real' cpp if you're on Linux/Unix) and "fake" libc includes
# to parse a file that includes standard C headers.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
import sys
......
......@@ -5,7 +5,7 @@
# of 'cpp'. The same can be achieved with Clang instead of gcc. If you have
# Clang installed, simply replace 'gcc' with 'clang' here.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-------------------------------------------------------------------------------
import sys
......
Metadata-Version: 1.1
Name: pycparser
Version: 2.17
Version: 2.18
Summary: C parser in Python
Home-page: https://github.com/eliben/pycparser
Author: Eli Bendersky
......
......@@ -5,11 +5,14 @@ README.rst
setup.cfg
setup.py
examples/c-to-c.py
examples/c_json.py
examples/cdecl.py
examples/dump_ast.py
examples/explore_ast.py
examples/func_calls.py
examples/func_defs.py
examples/rewrite_ast.py
examples/serialize_ast.py
examples/using_cpp_libc.py
examples/using_gcc_E_libc.py
examples/c_files/funky.c
......@@ -26,7 +29,9 @@ pycparser/c_ast.py
pycparser/c_generator.py
pycparser/c_lexer.py
pycparser/c_parser.py
pycparser/lextab.py
pycparser/plyparser.py
pycparser/yacctab.py
pycparser.egg-info/PKG-INFO
pycparser.egg-info/SOURCES.txt
pycparser.egg-info/dependency_links.txt
......@@ -118,10 +123,12 @@ utils/fake_libc_include/utmp.h
utils/fake_libc_include/wchar.h
utils/fake_libc_include/wctype.h
utils/fake_libc_include/zlib.h
utils/fake_libc_include/X11/Xlib.h
utils/fake_libc_include/arpa/inet.h
utils/fake_libc_include/asm-generic/int-ll64.h
utils/fake_libc_include/linux/socket.h
utils/fake_libc_include/linux/version.h
utils/fake_libc_include/mir_toolkit/client_types.h
utils/fake_libc_include/netinet/in.h
utils/fake_libc_include/netinet/tcp.h
utils/fake_libc_include/openssl/err.h
......@@ -142,4 +149,5 @@ utils/fake_libc_include/sys/types.h
utils/fake_libc_include/sys/uio.h
utils/fake_libc_include/sys/un.h
utils/fake_libc_include/sys/utsname.h
utils/fake_libc_include/sys/wait.h
\ No newline at end of file
utils/fake_libc_include/sys/wait.h
utils/fake_libc_include/xcb/xcb.h
\ No newline at end of file
......@@ -4,11 +4,11 @@
# This package file exports some convenience functions for
# interacting with pycparser
#
# Copyright (C) 2008-2016, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
__all__ = ['c_lexer', 'c_parser', 'c_ast']
__version__ = '2.17'
__version__ = '2.18'
from subprocess import Popen, PIPE
from .c_parser import CParser
......
......@@ -7,7 +7,7 @@
# The design of this module was inspired by astgen.py from the
# Python 2.5 code-base.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
import pprint
......@@ -150,7 +150,7 @@ r'''#-----------------------------------------------------------------
#
# AST Node classes.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
......
......@@ -6,7 +6,7 @@
# Also generates AST code from the configuration file.
# Should be called from the pycparser directory.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
......
......@@ -9,7 +9,7 @@
# <name>** - a sequence of child nodes
# <name> - an attribute
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
......
......@@ -3,7 +3,7 @@
#
# Some utilities used by the parser to create a friendlier AST.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#------------------------------------------------------------------------------
......
......@@ -11,7 +11,7 @@
#
# AST Node classes.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#-----------------------------------------------------------------
......
......@@ -3,7 +3,7 @@
#
# C code generator from pycparser AST nodes.
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#------------------------------------------------------------------------------
from . import c_ast
......@@ -178,6 +178,10 @@ class CGenerator(object):
s += self._make_indent() + '}\n'
return s
def visit_CompoundLiteral(self, n):
return '(' + self.visit(n.type) + '){' + self.visit(n.init) + '}'
def visit_EmptyStatement(self, n):
return ';'
......
......@@ -3,14 +3,14 @@
#
# CLexer class: lexer for the C language
#
# Copyright (C) 2008-2015, Eli Bendersky
# Eli Bendersky [http://eli.thegreenplace.net]
# License: BSD
#------------------------------------------------------------------------------
import re
import sys
from ply import lex
from ply.lex import TOKEN
from .ply import lex
from .ply.lex import TOKEN
class CLexer(object):
......@@ -52,8 +52,8 @@ class CLexer(object):
# Allow either "# line" or "# <num>" to support GCC's
# cpp output
#
self.line_pattern = re.compile('([ \t]*line\W)|([ \t]*\d+)')
self.pragma_pattern = re.compile('[ \t]*pragma\W')
self.line_pattern = re.compile(r'([ \t]*line\W)|([ \t]*\d+)')
self.pragma_pattern = re.compile(r'[ \t]*pragma\W')
def build(self, **kwargs):
""" Builds the lexer from the specification. Must be
......@@ -106,7 +106,7 @@ class CLexer(object):
'REGISTER', 'OFFSETOF',
'RESTRICT', 'RETURN', 'SHORT', 'SIGNED', 'SIZEOF', 'STATIC', 'STRUCT',
'SWITCH', 'TYPEDEF', 'UNION', 'UNSIGNED', 'VOID',
'VOLATILE', 'WHILE',
'VOLATILE', 'WHILE', '__INT128',
)
keyword_map = {}
......
This diff is collapsed.
# lextab.py. This file automatically created by PLY (version 3.8). Don't edit!
_tabversion = '3.8'
_lextokens = set(['VOID', 'LBRACKET', 'WCHAR_CONST', 'FLOAT_CONST', 'MINUS', 'RPAREN', 'LONG', 'PLUS', 'ELLIPSIS', 'GT', 'GOTO', 'ENUM', 'PERIOD', 'GE', 'INT_CONST_DEC', 'ARROW', 'HEX_FLOAT_CONST', 'DOUBLE', 'MINUSEQUAL', 'INT_CONST_OCT', 'TIMESEQUAL', 'OR', 'SHORT', 'RETURN', 'RSHIFTEQUAL', 'RESTRICT', 'STATIC', 'SIZEOF', 'UNSIGNED', 'UNION', 'COLON', 'WSTRING_LITERAL', 'DIVIDE', 'FOR', 'PLUSPLUS', 'EQUALS', 'ELSE', 'INLINE', 'EQ', 'AND', 'TYPEID', 'LBRACE', 'PPHASH', 'INT', 'SIGNED', 'CONTINUE', 'NOT', 'OREQUAL', 'MOD', 'RSHIFT', 'DEFAULT', 'CHAR', 'WHILE', 'DIVEQUAL', 'EXTERN', 'CASE', 'LAND', 'REGISTER', 'MODEQUAL', 'NE', 'SWITCH', 'INT_CONST_HEX', '_COMPLEX', 'PPPRAGMASTR', 'PLUSEQUAL', 'STRUCT', 'CONDOP', 'BREAK', 'VOLATILE', 'PPPRAGMA', 'ANDEQUAL', 'INT_CONST_BIN', 'DO', 'LNOT', 'CONST', 'LOR', 'CHAR_CONST', 'LSHIFT', 'RBRACE', '_BOOL', 'LE', 'SEMI', 'LT', 'COMMA', 'OFFSETOF', 'TYPEDEF', 'XOR', 'AUTO', 'TIMES', 'LPAREN', 'MINUSMINUS', 'ID', 'IF', 'STRING_LITERAL', 'FLOAT', 'XOREQUAL', 'LSHIFTEQUAL', 'RBRACKET'])
_lexreflags = 0
# lextab.py. This file automatically created by PLY (version 3.10). Don't edit!
_tabversion = '3.10'
_lextokens = set(('VOID', 'LBRACKET', 'WCHAR_CONST', 'FLOAT_CONST', 'MINUS', 'RPAREN', 'LONG', 'PLUS', 'ELLIPSIS', 'GT', 'GOTO', 'ENUM', 'PERIOD', 'GE', 'INT_CONST_DEC', 'ARROW', '__INT128', 'HEX_FLOAT_CONST', 'DOUBLE', 'MINUSEQUAL', 'INT_CONST_OCT', 'TIMESEQUAL', 'OR', 'SHORT', 'RETURN', 'RSHIFTEQUAL', 'RESTRICT', 'STATIC', 'SIZEOF', 'UNSIGNED', 'UNION', 'COLON', 'WSTRING_LITERAL', 'DIVIDE', 'FOR', 'PLUSPLUS', 'EQUALS', 'ELSE', 'INLINE', 'EQ', 'AND', 'TYPEID', 'LBRACE', 'PPHASH', 'INT', 'SIGNED', 'CONTINUE', 'NOT', 'OREQUAL', 'MOD', 'RSHIFT', 'DEFAULT', 'CHAR', 'WHILE', 'DIVEQUAL', 'EXTERN', 'CASE', 'LAND', 'REGISTER', 'MODEQUAL', 'NE', 'SWITCH', 'INT_CONST_HEX', '_COMPLEX', 'PPPRAGMASTR', 'PLUSEQUAL', 'STRUCT', 'CONDOP', 'BREAK', 'VOLATILE', 'PPPRAGMA', 'ANDEQUAL', 'INT_CONST_BIN', 'DO', 'LNOT', 'CONST', 'LOR', 'CHAR_CONST', 'LSHIFT', 'RBRACE', '_BOOL', 'LE', 'SEMI', 'LT', 'COMMA', 'OFFSETOF', 'TYPEDEF', 'XOR', 'AUTO', 'TIMES', 'LPAREN', 'MINUSMINUS', 'ID', 'IF', 'STRING_LITERAL', 'FLOAT', 'XOREQUAL', 'LSHIFTEQUAL', 'RBRACKET'))
_lexreflags = 64
_lexliterals = ''
_lexstateinfo = {'ppline': 'exclusive', 'pppragma': 'exclusive', 'INITIAL': 'inclusive'}
_lexstatere = {'ppline': [('(?P<t_ppline_FILENAME>"([^"\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))*")|(?P<t_ppline_LINE_NUMBER>(0(([uU]ll)|([uU]LL)|(ll[uU]?)|(LL[uU]?)|([uU][lL])|([lL][uU]?)|[uU])?)|([1-9][0-9]*(([uU]ll)|([uU]LL)|(ll[uU]?)|(LL[uU]?)|([uU][lL])|([lL][uU]?)|[uU])?))|(?P<t_ppline_NEWLINE>\\n)|(?P<t_ppline_PPLINE>line)', [None, ('t_ppline_FILENAME', 'FILENAME'), None, None, None, None, None, None, ('t_ppline_LINE_NUMBER', 'LINE_NUMBER'), None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, ('t_ppline_NEWLINE', 'NEWLINE'), ('t_ppline_PPLINE', 'PPLINE')])], 'pppragma': [('(?P<t_pppragma_NEWLINE>\\n)|(?P<t_pppragma_PPPRAGMA>pragma)|(?P<t_pppragma_STR>.+)', [None, ('t_pppragma_NEWLINE', 'NEWLINE'), ('t_pppragma_PPPRAGMA', 'PPPRAGMA'), ('t_pppragma_STR', 'STR')])], 'INITIAL': [('(?P<t_PPHASH>[ \\t]*\\#)|(?P<t_NEWLINE>\\n+)|(?P<t_LBRACE>\\{)|(?P<t_RBRACE>\\})|(?P<t_FLOAT_CONST>((((([0-9]*\\.[0-9]+)|([0-9]+\\.))([eE][-+]?[0-9]+)?)|([0-9]+([eE][-+]?[0-9]+)))[FfLl]?))|(?P<t_HEX_FLOAT_CONST>(0[xX]([0-9a-fA-F]+|((([0-9a-fA-F]+)?\\.[0-9a-fA-F]+)|([0-9a-fA-F]+\\.)))([pP][+-]?[0-9]+)[FfLl]?))|(?P<t_INT_CONST_HEX>0[xX][0-9a-fA-F]+(([uU]ll)|([uU]LL)|(ll[uU]?)|(LL[uU]?)|([uU][lL])|([lL][uU]?)|[uU])?)', [None, ('t_PPHASH', 'PPHASH'), ('t_NEWLINE', 'NEWLINE'), ('t_LBRACE', 'LBRACE'), ('t_RBRACE', 'RBRACE'), ('t_FLOAT_CONST', 'FLOAT_CONST'), None, None, None, None, None, None, None, None, None, ('t_HEX_FLOAT_CONST', 'HEX_FLOAT_CONST'), None, None, None, None, None, None, None, ('t_INT_CONST_HEX', 'INT_CONST_HEX')]), ('(?P<t_INT_CONST_BIN>0[bB][01]+(([uU]ll)|([uU]LL)|(ll[uU]?)|(LL[uU]?)|([uU][lL])|([lL][uU]?)|[uU])?)|(?P<t_BAD_CONST_OCT>0[0-7]*[89])|(?P<t_INT_CONST_OCT>0[0-7]*(([uU]ll)|([uU]LL)|(ll[uU]?)|(LL[uU]?)|([uU][lL])|([lL][uU]?)|[uU])?)|(?P<t_INT_CONST_DEC>(0(([uU]ll)|([uU]LL)|(ll[uU]?)|(LL[uU]?)|([uU][lL])|([lL][uU]?)|[uU])?)|([1-9][0-9]*(([uU]ll)|([uU]LL)|(ll[uU]?)|(LL[uU]?)|([uU][lL])|([lL][uU]?)|[uU])?))|(?P<t_CHAR_CONST>\'([^\'\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))\')|(?P<t_WCHAR_CONST>L\'([^\'\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))\')|(?P<t_UNMATCHED_QUOTE>(\'([^\'\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))*\\n)|(\'([^\'\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))*$))|(?P<t_BAD_CHAR_CONST>(\'([^\'\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))[^\'\n]+\')|(\'\')|(\'([\\\\][^a-zA-Z._~^!=&\\^\\-\\\\?\'"x0-7])[^\'\\n]*\'))', [None, ('t_INT_CONST_BIN', 'INT_CONST_BIN'), None, None, None, None, None, None, None, ('t_BAD_CONST_OCT', 'BAD_CONST_OCT'), ('t_INT_CONST_OCT', 'INT_CONST_OCT'), None, None, None, None, None, None, None, ('t_INT_CONST_DEC', 'INT_CONST_DEC'), None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, ('t_CHAR_CONST', 'CHAR_CONST'), None, None, None, None, None, None, ('t_WCHAR_CONST', 'WCHAR_CONST'), None, None, None, None, None, None, ('t_UNMATCHED_QUOTE', 'UNMATCHED_QUOTE'), None, None, None, None, None, None, None, None, None, None, None, None, None, None, ('t_BAD_CHAR_CONST', 'BAD_CHAR_CONST')]), ('(?P<t_WSTRING_LITERAL>L"([^"\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))*")|(?P<t_BAD_STRING_LITERAL>"([^"\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))*?([\\\\][^a-zA-Z._~^!=&\\^\\-\\\\?\'"x0-7])([^"\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))*")|(?P<t_ID>[a-zA-Z_$][0-9a-zA-Z_$]*)|(?P<t_STRING_LITERAL>"([^"\\\\\\n]|(\\\\(([a-zA-Z._~!=&\\^\\-\\\\?\'"])|(\\d+)|(x[0-9a-fA-F]+))))*")|(?P<t_ELLIPSIS>\\.\\.\\.)|(?P<t_PLUSPLUS>\\+\\+)|(?P<t_LOR>\\|\\|)|(?P<t_XOREQUAL>\\^=)|(?P<t_OREQUAL>\\|=)|(?P<t_LSHIFTEQUAL><<=)|(?P<t_RSHIFTEQUAL>>>=)|(?P<t_PLUSEQUAL>\\+=)|(?P<t_TIMESEQUAL>\\*=)|(?P<t_PLUS>\\+)|(?P<t_MODEQUAL>%=)|(?P<t_DIVEQUAL>/=)', [None, ('t_WSTRING_LITERAL', 'WSTRING_LITERAL'), None, None, None, None, None, None, ('t_BAD_STRING_LITERAL', 'BAD_STRING_LITERAL'), None, None, None, None, None, None, None, None, None, None, None, None, None, ('t_ID', 'ID'), (None, 'STRING_LITERAL'), None, None, None, None, None, None, (None, 'ELLIPSIS'), (None, 'PLUSPLUS'), (None, 'LOR'), (None, 'XOREQUAL'), (None, 'OREQUAL'), (None, 'LSHIFTEQUAL'), (None, 'RSHIFTEQUAL'), (None, 'PLUSEQUAL'), (None, 'TIMESEQUAL'), (None, 'PLUS'), (None, 'MODEQUAL'), (None, 'DIVEQUAL')]), ('(?P<t_RBRACKET>\\])|(?P<t_CONDOP>\\?)|(?P<t_XOR>\\^)|(?P<t_LSHIFT><<)|(?P<t_LE><=)|(?P<t_LPAREN>\\()|(?P<t_ARROW>->)|(?P<t_EQ>==)|(?P<t_NE>!=)|(?P<t_MINUSMINUS>--)|(?P<t_OR>\\|)|(?P<t_TIMES>\\*)|(?P<t_LBRACKET>\\[)|(?P<t_GE>>=)|(?P<t_RPAREN>\\))|(?P<t_LAND>&&)|(?P<t_RSHIFT>>>)|(?P<t_MINUSEQUAL>-=)|(?P<t_PERIOD>\\.)|(?P<t_ANDEQUAL>&=)|(?P<t_EQUALS>=)|(?P<t_LT><)|(?P<t_COMMA>,)|(?P<t_DIVIDE>/)|(?P<t_AND>&)|(?P<t_MOD>%)|(?P<t_SEMI>;)|(?P<t_MINUS>-)|(?P<t_GT>>)|(?P<t_COLON>:)|(?P<t_NOT>~)|(?P<t_LNOT>!)', [None, (None, 'RBRACKET'), (None, 'CONDOP'), (None, 'XOR'), (None, 'LSHIFT'), (None, 'LE'), (None, 'LPAREN'), (None, 'ARROW'), (None, 'EQ'), (None, 'NE'), (None, 'MINUSMINUS'), (None, 'OR'), (None, 'TIMES'), (None, 'LBRACKET'), (None, 'GE'), (None, 'RPAREN'), (None, 'LAND'), (None, 'RSHIFT'), (None, 'MINUSEQUAL'), (None, 'PERIOD'), (None, 'ANDEQUAL'), (None, 'EQUALS'), (None, 'LT'), (None, 'COMMA'), (None, 'DIVIDE'), (None, 'AND'), (None, 'MOD'), (None, 'SEMI'), (None, 'MINUS'), (None, 'GT'), (None, 'COLON'), (None, 'NOT'), (None, 'LNOT')])]}
......
# PLY package
# Author: David Beazley (dave@dabeaz.com)
__version__ = '3.7'
__version__ = '3.9'
__all__ = ['lex','yacc']
......@@ -2,13 +2,22 @@
# cpp.py
#
# Author: David Beazley (http://www.dabeaz.com)
# Copyright (C) 2007
# Copyright (C) 2017
# All rights reserved
#
# This module implements an ANSI-C style lexical preprocessor for PLY.
# This module implements an ANSI-C style lexical preprocessor for PLY.
# -----------------------------------------------------------------------------
from __future__ import generators
import sys
# Some Python 3 compatibility shims
if sys.version_info.major < 3:
STRING_TYPES = (str, unicode)
else:
STRING_TYPES = str
xrange = range
# -----------------------------------------------------------------------------
# Default preprocessor lexer definitions. These tokens are enough to get
# a basic preprocessor working. Other modules may import these if they want
......@@ -68,7 +77,8 @@ def t_CPP_COMMENT2(t):
r'(//.*?(\n|$))'
# replace with '/n'
t.type = 'CPP_WS'; t.value = '\n'
return t
def t_error(t):
t.type = t.value[0]
t.value = t.value[0]
......@@ -82,8 +92,8 @@ import os.path
# -----------------------------------------------------------------------------
# trigraph()
#
# Given an input string, this function replaces all trigraph sequences.
#
# Given an input string, this function replaces all trigraph sequences.
# The following mapping is used:
#
# ??= #
......@@ -253,7 +263,7 @@ class Preprocessor(object):
# ----------------------------------------------------------------------
# add_path()
#
# Adds a search path to the preprocessor.
# Adds a search path to the preprocessor.
# ----------------------------------------------------------------------
def add_path(self,path):
......@@ -297,7 +307,7 @@ class Preprocessor(object):
# ----------------------------------------------------------------------
# tokenstrip()
#
#
# Remove leading/trailing whitespace tokens from a token list
# ----------------------------------------------------------------------
......@@ -323,7 +333,7 @@ class Preprocessor(object):
# argument. Each argument is represented by a list of tokens.
#
# When collecting arguments, leading and trailing whitespace is removed
# from each argument.
# from each argument.
#
# This function properly handles nested parenthesis and commas---these do not
# define new arguments.
......@@ -335,7 +345,7 @@ class Preprocessor(object):
current_arg = []
nesting = 1
tokenlen = len(tokenlist)
# Search for the opening '('.
i = 0
while (i < tokenlen) and (tokenlist[i].type in self.t_WS):
......@@ -369,7 +379,7 @@ class Preprocessor(object):
else:
current_arg.append(t)
i += 1
# Missing end argument
self.error(self.source,tokenlist[-1].lineno,"Missing ')' in macro arguments")
return 0, [],[]
......@@ -381,9 +391,9 @@ class Preprocessor(object):
# This is used to speed up macro expansion later on---we'll know
# right away where to apply patches to the value to form the expansion
# ----------------------------------------------------------------------
def macro_prescan(self,macro):
macro.patch = [] # Standard macro arguments
macro.patch = [] # Standard macro arguments
macro.str_patch = [] # String conversion expansion
macro.var_comma_patch = [] # Variadic macro comma patch
i = 0
......@@ -430,7 +440,7 @@ class Preprocessor(object):
rep = [copy.copy(_x) for _x in macro.value]
# Make string expansion patches. These do not alter the length of the replacement sequence
str_expansion = {}
for argnum, i in macro.str_patch:
if argnum not in str_expansion:
......@@ -448,7 +458,7 @@ class Preprocessor(object):
# Make all other patches. The order of these matters. It is assumed that the patch list
# has been sorted in reverse order of patch location since replacements will cause the
# size of the replacement sequence to expand from the patch point.
expanded = { }
for ptype, argnum, i in macro.patch:
# Concatenation. Argument is left unexpanded
......@@ -485,7 +495,7 @@ class Preprocessor(object):
if t.value in self.macros and t.value not in expanded:
# Yes, we found a macro match
expanded[t.value] = True
m = self.macros[t.value]
if not m.arglist:
# A simple macro
......@@ -517,7 +527,7 @@ class Preprocessor(object):
else:
args[len(m.arglist)-1] = tokens[j+positions[len(m.arglist)-1]:j+tokcount-1]
del args[len(m.arglist):]
# Get macro replacement text
rep = self.macro_expand_args(m,args)
rep = self.expand_macros(rep,expanded)
......@@ -530,13 +540,13 @@ class Preprocessor(object):
elif t.value == '__LINE__':
t.type = self.t_INTEGER
t.value = self.t_INTEGER_TYPE(t.lineno)
i += 1
return tokens
# ----------------------------------------------------------------------
# ----------------------------------------------------------------------
# evalexpr()
#
#
# Evaluate an expression token sequence for the purposes of evaluating
# integral expressions.
# ----------------------------------------------------------------------
......@@ -583,14 +593,14 @@ class Preprocessor(object):
tokens[i].value = str(tokens[i].value)
while tokens[i].value[-1] not in "0123456789abcdefABCDEF":
tokens[i].value = tokens[i].value[:-1]
expr = "".join([str(x.value) for x in tokens])
expr = expr.replace("&&"," and ")
expr = expr.replace("||"," or ")
expr = expr.replace("!"," not ")
try:
result = eval(expr)
except StandardError:
except Exception:
self.error(self.source,tokens[0].lineno,"Couldn't evaluate expression")
result = 0
return result
......@@ -608,7 +618,7 @@ class Preprocessor(object):
if not source:
source = ""
self.define("__FILE__ \"%s\"" % source)
self.source = source
......@@ -627,7 +637,7 @@ class Preprocessor(object):
for tok in x:
if tok.type in self.t_WS and '\n' in tok.value:
chunk.append(tok)
dirtokens = self.tokenstrip(x[i+1:])
if dirtokens:
name = dirtokens[0].value
......@@ -635,7 +645,7 @@ class Preprocessor(object):
else:
name = ""
args = []
if name == 'define':
if enable:
for tok in self.expand_macros(chunk):
......@@ -695,7 +705,7 @@ class Preprocessor(object):
iftrigger = True
else:
self.error(self.source,dirtokens[0].lineno,"Misplaced #elif")
elif name == 'else':
if ifstack:
if ifstack[-1][0]:
......@@ -781,7 +791,7 @@ class Preprocessor(object):
# ----------------------------------------------------------------------
def define(self,tokens):
if isinstance(tokens,(str,unicode)):
if isinstance(tokens,STRING_TYPES):
tokens = self.tokenize(tokens)
linetok = tokens
......@@ -865,7 +875,7 @@ class Preprocessor(object):
def parse(self,input,source=None,ignore={}):
self.ignore = ignore
self.parser = self.parsegen(input,source)
# ----------------------------------------------------------------------
# token()
#
......@@ -895,14 +905,3 @@ if __name__ == '__main__':
tok = p.token()
if not tok: break
print(p.source, tok)
# -----------------------------------------------------------------------------
# ply: lex.py
#
# Copyright (C) 2001-2015,
# Copyright (C) 2001-2017
# David M. Beazley (Dabeaz LLC)
# All rights reserved.
#
......@@ -31,8 +31,8 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# -----------------------------------------------------------------------------
__version__ = '3.8'
__tabversion__ = '3.8'
__version__ = '3.10'
__tabversion__ = '3.10'
import re
import sys
......@@ -179,12 +179,12 @@ class Lexer:
with open(filename, 'w') as tf:
tf.write('# %s.py. This file automatically created by PLY (version %s). Don\'t edit!\n' % (basetabmodule, __version__))
tf.write('_tabversion = %s\n' % repr(__tabversion__))
tf.write('_lextokens = %s\n' % repr(self.lextokens))
tf.write('_lextokens = set(%s)\n' % repr(tuple(self.lextokens)))
tf.write('_lexreflags = %s\n' % repr(self.lexreflags))
tf.write('_lexliterals = %s\n' % repr(self.lexliterals))
tf.write('_lexstateinfo = %s\n' % repr(self.lexstateinfo))
# Rewrite the lexstatere table, replacing function objects with function names
# Rewrite the lexstatere table, replacing function objects with function names
tabre = {}
for statename, lre in self.lexstatere.items():
titem = []
......@@ -230,7 +230,7 @@ class Lexer:
titem = []
txtitem = []
for pat, func_name in lre:
titem.append((re.compile(pat, lextab._lexreflags | re.VERBOSE), _names_to_funcs(func_name, fdict)))
titem.append((re.compile(pat, lextab._lexreflags), _names_to_funcs(func_name, fdict)))
self.lexstatere[statename] = titem
self.lexstateretext[statename] = txtitem
......@@ -495,7 +495,7 @@ def _form_master_re(relist, reflags, ldict, toknames):
return []
regex = '|'.join(relist)
try:
lexre = re.compile(regex, re.VERBOSE | reflags)
lexre = re.compile(regex, reflags)
# Build the index to function map for the matching engine
lexindexfunc = [None] * (max(lexre.groupindex.values()) + 1)
......@@ -536,7 +536,7 @@ def _statetoken(s, names):
for i, part in enumerate(parts[1:], 1):
if part not in names and part != 'ANY':
break
if i > 1:
states = tuple(parts[1:i])
else:
......@@ -758,7 +758,7 @@ class LexerReflect(object):
continue
try:
c = re.compile('(?P<%s>%s)' % (fname, _get_regex(f)), re.VERBOSE | self.reflags)
c = re.compile('(?P<%s>%s)' % (fname, _get_regex(f)), self.reflags)
if c