From 101671ae44e1686680c80cd07b452aabeb88fb63 Mon Sep 17 00:00:00 2001 From: goodger Date: Sat, 20 Apr 2002 03:01:52 +0000 Subject: Initial revision git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@18 929543f6-e4f2-0310-98a6-ba3bd3dd1d04 --- docs/dev/pysource.txt | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 docs/dev/pysource.txt (limited to 'docs/dev/pysource.txt') diff --git a/docs/dev/pysource.txt b/docs/dev/pysource.txt new file mode 100644 index 000000000..a75fd1d4d --- /dev/null +++ b/docs/dev/pysource.txt @@ -0,0 +1,173 @@ +====================== + Python Source Reader +====================== +:Author: David Goodger +:Contact: goodger@users.sourceforge.net +:Revision: $Revision$ +:Date: $Date$ + +This document explores issues around extracting and processing +docstrings from Python modules. + +For definitive element hierarchy details, see the "Python Plaintext +Document Interface DTD" XML document type definition, pysource.dtd_ +(which modifies the generic docutils.dtd_). Descriptions below list +'DTD elements' (XML 'generic identifiers' or tag names) corresponding +to syntax constructs. + + +.. contents:: + + +Python Source Reader +==================== + +The Python Source Reader ("PySource") model that's evolving in my mind +goes something like this: + +1. Extract the docstring/namespace [#]_ tree from the module(s) and/or + package(s). + + .. [#] See `Docstring Extractor`_ above. + +2. Run the parser on each docstring in turn, producing a forest of + doctrees (per nodes.py). + +3. Join the docstring trees together into a single tree, running + transforms: + + - merge hyperlinks + - merge namespaces + - create various sections like "Module Attributes", "Functions", + "Classes", "Class Attributes", etc.; see spec/ppdi.dtd + - convert the above special sections to ordinary doctree nodes + +4. Run transforms on the combined doctree. Examples: resolving + cross-references/hyperlinks (including interpreted text on Python + identifiers); footnote auto-numbering; first field list -> + bibliographic elements. + + (Or should step 4's transforms come before step 3?) + +5. Pass the resulting unified tree to the writer/builder. + +I've had trouble reconciling the roles of input parser and output +writer with the idea of modes ("readers" or "directors"). Does the +mode govern the tranformation of the input, the output, or both? +Perhaps the mode should be split into two. + +For example, say the source of our input is a Python module. Our +"input mode" should be the "Python Source Reader". It discovers (from +``__docformat__``) that the input parser is "reStructuredText". If we +want HTML, we'll specify the "HTML" output formatter. But there's a +piece missing. What *kind* or *style* of HTML output do we want? +PyDoc-style, LibRefMan style, etc. (many people will want to specify +and control their own style). Is the output style specific to a +particular output format (XML, HTML, etc.)? Is the style specific to +the input mode? Or can/should they be independent? + +I envision interaction between the input parser, an "input mode" , and +the output formatter. The same intermediate data format would be used +between each of these, being transformed as it progresses. + + +Docstring Extractor +=================== + +We need code that scans a parsed Python module, and returns an ordered +tree containing the names, docstrings (including attribute and +additional docstrings), and additional info (in parentheses below) of +all of the following objects: + +- packages +- modules +- module attributes (+ values) +- classes (+ inheritance) +- class attributes (+ values) +- instance attributes (+ values) +- methods (+ formal parameters & defaults) +- functions (+ formal parameters & defaults) + +(Extract comments too? For example, comments at the start of a module +would be a good place for bibliographic field lists.) + +In order to evaluate interpreted text cross-references, namespaces for +each of the above will also be required. + +See python-dev/docstring-develop thread "AST mining", started on +2001-08-14. + + +Interpreted Text +================ + +DTD elements: package, module, class, method, function, +module_attribute, class_attribute, instance_attribute, variable, +parameter, type, exception_class, warning_class. + +In Python docstrings, interpreted text is used to classify and mark up +program identifiers, such as the names of variables, functions, +classes, and modules. If the identifier alone is given, its role is +inferred implicitly according to the Python namespace lookup rules. +For functions and methods (even when dynamically assigned), +parentheses ('()') may be included:: + + This function uses `another()` to do its work. + +For class, instance and module attributes, dotted identifiers are used +when necessary:: + + class Keeper(Storer): + + """ + Extend `Storer`. Class attribute `instances` keeps track of + the number of `Keeper` objects instantiated. + """ + + instances = 0 + """How many `Keeper` objects are there?""" + + def __init__(self): + """ + Extend `Storer.__init__()` to keep track of instances. + + Keep count in `self.instances` and data in `self.data`. + """ + Storer.__init__(self) + self.instances += 1 + + self.data = [] + """Store data in a list, most recent last.""" + + def storedata(self, data): + """ + Extend `Storer.storedata()`; append new `data` to a list + (in `self.data`). + """ + self.data = data + +To classify identifiers explicitly, the role is given along with the +identifier in either prefix or suffix form:: + + Use :method:`Keeper.storedata` to store the object's data in + `Keeper.data`:instance_attribute:. + +The role may be one of 'package', 'module', 'class', 'method', +'function', 'module_attribute', 'class_attribute', +'instance_attribute', 'variable', 'parameter', 'type', +'exception_class', 'exception', 'warning_class', or 'warning'. Other +roles may be defined. + +.. _reStructuredText Markup Specification: + http://docutils.sourceforge.net/spec/rst/reStructuredText.html +.. _Docutils: http://docutils.sourceforge.net/ +.. _pysource.dtd: http://docutils.sourceforge.net/spec/pysource.dtd +.. _docutils.dtd: http://docutils.sourceforge.net/spec/docutils.dtd + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + fill-column: 70 + End: -- cgit v1.2.1 From 6b21c047594ea25b4fa75f903e9b315859735116 Mon Sep 17 00:00:00 2001 From: goodger Date: Sun, 5 May 2002 17:06:02 +0000 Subject: fixed hyperlinks git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@97 929543f6-e4f2-0310-98a6-ba3bd3dd1d04 --- docs/dev/pysource.txt | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) (limited to 'docs/dev/pysource.txt') diff --git a/docs/dev/pysource.txt b/docs/dev/pysource.txt index a75fd1d4d..93fba9420 100644 --- a/docs/dev/pysource.txt +++ b/docs/dev/pysource.txt @@ -19,8 +19,8 @@ to syntax constructs. .. contents:: -Python Source Reader -==================== +Model +===== The Python Source Reader ("PySource") model that's evolving in my mind goes something like this: @@ -158,9 +158,6 @@ The role may be one of 'package', 'module', 'class', 'method', 'exception_class', 'exception', 'warning_class', or 'warning'. Other roles may be defined. -.. _reStructuredText Markup Specification: - http://docutils.sourceforge.net/spec/rst/reStructuredText.html -.. _Docutils: http://docutils.sourceforge.net/ .. _pysource.dtd: http://docutils.sourceforge.net/spec/pysource.dtd .. _docutils.dtd: http://docutils.sourceforge.net/spec/docutils.dtd -- cgit v1.2.1 From 813520af216474ec5c97efb2b1809e38aa72b5d1 Mon Sep 17 00:00:00 2001 From: goodger Date: Sun, 29 Dec 2002 18:38:06 +0000 Subject: updated git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@1046 929543f6-e4f2-0310-98a6-ba3bd3dd1d04 --- docs/dev/pysource.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/dev/pysource.txt') diff --git a/docs/dev/pysource.txt b/docs/dev/pysource.txt index 93fba9420..1b5e629f7 100644 --- a/docs/dev/pysource.txt +++ b/docs/dev/pysource.txt @@ -28,7 +28,7 @@ goes something like this: 1. Extract the docstring/namespace [#]_ tree from the module(s) and/or package(s). - .. [#] See `Docstring Extractor`_ above. + .. [#] See `Docstring Extractor`_ below. 2. Run the parser on each docstring in turn, producing a forest of doctrees (per nodes.py). -- cgit v1.2.1 From 13681f75fbe4e198808988aa4dc962524e89fbcf Mon Sep 17 00:00:00 2001 From: goodger Date: Fri, 10 Jan 2003 02:25:51 +0000 Subject: updated git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@1090 929543f6-e4f2-0310-98a6-ba3bd3dd1d04 --- docs/dev/pysource.txt | 41 ----------------------------------------- 1 file changed, 41 deletions(-) (limited to 'docs/dev/pysource.txt') diff --git a/docs/dev/pysource.txt b/docs/dev/pysource.txt index 1b5e629f7..e1df7fb52 100644 --- a/docs/dev/pysource.txt +++ b/docs/dev/pysource.txt @@ -105,47 +105,6 @@ DTD elements: package, module, class, method, function, module_attribute, class_attribute, instance_attribute, variable, parameter, type, exception_class, warning_class. -In Python docstrings, interpreted text is used to classify and mark up -program identifiers, such as the names of variables, functions, -classes, and modules. If the identifier alone is given, its role is -inferred implicitly according to the Python namespace lookup rules. -For functions and methods (even when dynamically assigned), -parentheses ('()') may be included:: - - This function uses `another()` to do its work. - -For class, instance and module attributes, dotted identifiers are used -when necessary:: - - class Keeper(Storer): - - """ - Extend `Storer`. Class attribute `instances` keeps track of - the number of `Keeper` objects instantiated. - """ - - instances = 0 - """How many `Keeper` objects are there?""" - - def __init__(self): - """ - Extend `Storer.__init__()` to keep track of instances. - - Keep count in `self.instances` and data in `self.data`. - """ - Storer.__init__(self) - self.instances += 1 - - self.data = [] - """Store data in a list, most recent last.""" - - def storedata(self, data): - """ - Extend `Storer.storedata()`; append new `data` to a list - (in `self.data`). - """ - self.data = data - To classify identifiers explicitly, the role is given along with the identifier in either prefix or suffix form:: -- cgit v1.2.1 From 4cc798623cc64d7cd12d72105edb830ac7af07e6 Mon Sep 17 00:00:00 2001 From: goodger Date: Mon, 9 Jun 2003 21:26:31 +0000 Subject: clarification of public domain status git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@1409 929543f6-e4f2-0310-98a6-ba3bd3dd1d04 --- docs/dev/pysource.txt | 1 + 1 file changed, 1 insertion(+) (limited to 'docs/dev/pysource.txt') diff --git a/docs/dev/pysource.txt b/docs/dev/pysource.txt index e1df7fb52..ab677a004 100644 --- a/docs/dev/pysource.txt +++ b/docs/dev/pysource.txt @@ -5,6 +5,7 @@ :Contact: goodger@users.sourceforge.net :Revision: $Revision$ :Date: $Date$ +:Copyright: This document has been placed in the public domain. This document explores issues around extracting and processing docstrings from Python modules. -- cgit v1.2.1 From 02bca6cdf86ccf60250b1b66efe54361a665e5ea Mon Sep 17 00:00:00 2001 From: wiemann Date: Thu, 3 Jun 2004 19:35:53 +0000 Subject: updated git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@2206 929543f6-e4f2-0310-98a6-ba3bd3dd1d04 --- docs/dev/pysource.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'docs/dev/pysource.txt') diff --git a/docs/dev/pysource.txt b/docs/dev/pysource.txt index ab677a004..2336ebd54 100644 --- a/docs/dev/pysource.txt +++ b/docs/dev/pysource.txt @@ -118,8 +118,8 @@ The role may be one of 'package', 'module', 'class', 'method', 'exception_class', 'exception', 'warning_class', or 'warning'. Other roles may be defined. -.. _pysource.dtd: http://docutils.sourceforge.net/spec/pysource.dtd -.. _docutils.dtd: http://docutils.sourceforge.net/spec/docutils.dtd +.. _pysource.dtd: pysource.dtd +.. _docutils.dtd: ../ref/docutils.dtd .. -- cgit v1.2.1 From 6c18a0548558d3eefb532ef613a49533328dd357 Mon Sep 17 00:00:00 2001 From: wiemann Date: Sat, 5 Jun 2004 19:32:15 +0000 Subject: updated git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@2223 929543f6-e4f2-0310-98a6-ba3bd3dd1d04 --- docs/dev/pysource.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/dev/pysource.txt') diff --git a/docs/dev/pysource.txt b/docs/dev/pysource.txt index 2336ebd54..6f173a709 100644 --- a/docs/dev/pysource.txt +++ b/docs/dev/pysource.txt @@ -40,7 +40,7 @@ goes something like this: - merge hyperlinks - merge namespaces - create various sections like "Module Attributes", "Functions", - "Classes", "Class Attributes", etc.; see spec/ppdi.dtd + "Classes", "Class Attributes", etc.; see pysource.dtd_ - convert the above special sections to ordinary doctree nodes 4. Run transforms on the combined doctree. Examples: resolving -- cgit v1.2.1