Guide for new developers.

Documentation design.
Expand test case design.

Copied from Perforce
 Change: 195083

mps/code/comm.gmk

@@ -3,7 +3,7 @@
 # comm.gmk: COMMON GNUMAKEFILE FRAGMENT
 #
 # $Id$
-# Copyright (c) 2001-2016 Ravenbrook Limited.  See end of file for license.
+# Copyright (c) 2001-2018 Ravenbrook Limited.  See end of file for license.
 #
 # DESCRIPTION
 #
@@ -308,11 +308,7 @@ all: $(ALL_TARGETS)
 
 # == Automated test suites ==
 #
-# testrun = "smoke test", fast enough to run before every commit
+# See design.mps.tests.target.
-# testci = continuous integration tests, must be known good
-# testall = all test cases, for ensuring quality of a release
-# testansi = tests that run on the generic ("ANSI") platform
-# testpollnone = tests that run on the generic platform with CONFIG_POLL_NONE
 
 TEST_SUITES=testrun testci testall testansi testpollnone
 
@@ -714,7 +710,7 @@ find-puns: phony
 
 # C. COPYRIGHT AND LICENSE
 #
-# Copyright (c) 2001-2016 Ravenbrook Limited <http://www.ravenbrook.com/>.
+# Copyright (c) 2001-2018 Ravenbrook Limited <http://www.ravenbrook.com/>.
 # All rights reserved.  This is an open source license.  Contact
 # Ravenbrook for commercial licensing options.
 # 

mps/design/config.txt

@@ -164,9 +164,7 @@ predefined when compiling the module sources::
 
     CONFIG_VAR_<variety-code>
 
-The variety-codes are the letter code that appears after "variety." in
+_`.var`: The variety codes are as follows:
-the tag of the relevant variety document (see variety.*) converted to
-upper case. Currently (2012-09-03):
 
 _`.var.hot`: ``HOT``
 
@@ -621,7 +619,7 @@ Document History
 Copyright and License
 ---------------------
 
-Copyright © 2013-2016 Ravenbrook Limited <http://www.ravenbrook.com/>.
+Copyright © 2013-2018 Ravenbrook Limited <http://www.ravenbrook.com/>.
 All rights reserved. This is an open source license. Contact
 Ravenbrook for commercial licensing options.
 

mps/design/doc.txt

@@ -0,0 +1,149 @@
+.. mode: -*- rst -*-
+
+Documentation
+=============
+
+:Tag: design.mps.doc
+:Author: Gareth Rees
+:Date: 2018-09-18
+:Status: draft design
+:Revision: $Id$
+:Copyright: See section `Copyright and License`_.
+:Index terms: pair: documentation; design
+
+
+Introduction
+------------
+
+_`.intro`: This is the design of the documentation system for the
+Memory Pool System.
+
+_`.readership`: This document is intended for any MPS developer.
+
+
+Types
+-----
+
+_`.type`: The MPS has multiple types of documentation, suitable for
+different audiences.
+
+_`.type.comment`: Comments in the code provide information that is
+required in order for developers to make correct edits to nearby code.
+(Audience: MPS developers editing nearby code.)
+
+_`.type.design`: Design documentation lists requirements and explains
+how the code meets the requirements. (Audience: MPS developers working
+on a subsystem.)
+
+_`.type.devguide`: Developer guides provide general guidance for
+developers, not specific to any particular subsystem. (Audience: MPS
+developers generally.)
+
+_`.type.procedure`: Procedures list the steps for carrying out
+development tasks. (Audience: MPS developers who need to carry out
+particular tasks reliably.)
+
+_`.type.tutorial`: Tutorials describe how to use the MPS to meet
+client program requirements. (Audience: beginner client program
+developers.)
+
+_`.type.reference`: Reference documentation specifies the public
+features of the MPS. (Audience: expert client program developers.)
+
+_`.type.mmref`: The Memory Management Reference describes general
+principles of memory management, with cross-references to the MPS
+documentation. (Audience: the world.)
+
+
+Requirements
+------------
+
+_`.req.source`: Derived from [RB_2013-05-09]_.
+
+_`.req.easy`: It must be easy to read and write documentation must
+using standard text editors. Barriers to documentation must be low.
+
+_`.req.presentation`: It must be possible to process documentation
+into presentation formats, for example web pages.
+
+_`.req.single-source`: Documents must have a single source. Processing
+into other formats must be automatic and not depend on hand editing or
+maintaining parallel versions.
+
+_`.req.durable`: The format of documents should be supported for the
+foreseeable future. It must not require continual updating to keep up
+with changes to processing software.
+
+_`.req.design.ref`: It must be easy to reference points made in design
+documents from the code.
+
+_`.req.design.standalone`: Design documents must stand alone: they
+must not require particular software to make them readable or
+complete.
+
+
+Implementation
+--------------
+
+TODO
+
+
+
+References
+----------
+
+.. [RB_2013-05-09]
+   "MPS design document format and process";
+   Richard Brooksby; Ravenbrook Limited; 2013-05-09;
+   <https://info.ravenbrook.com/mail/2013/05/09/14-54-55/0/>.
+
+
+Document History
+----------------
+
+- 2018-09-18 GDR_ Created based on [RB_2013-05-09]_.
+
+.. _GDR: http://www.ravenbrook.com/consultants/gdr/
+
+
+Copyright and License
+---------------------
+
+Copyright © 2018 Ravenbrook Limited <http://www.ravenbrook.com/>.
+All rights reserved. This is an open source license. Contact
+Ravenbrook for commercial licensing options.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+#. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+
+#. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+#. Redistributions in any form must be accompanied by information on how
+   to obtain complete source code for this software and any
+   accompanying software that uses this software.  The source code must
+   either be included in the distribution or be available for no more than
+   the cost of distribution plus a nominal fee, and must be freely
+   redistributable under reasonable conditions.  For an executable file,
+   complete source code means the source code for all modules it contains.
+   It does not include source code for modules or files that typically
+   accompany the major components of the operating system on which the
+   executable file runs.
+
+**This software is provided by the copyright holders and contributors
+"as is" and any express or implied warranties, including, but not
+limited to, the implied warranties of merchantability, fitness for a
+particular purpose, or non-infringement, are disclaimed.  In no event
+shall the copyright holders and contributors be liable for any direct,
+indirect, incidental, special, exemplary, or consequential damages
+(including, but not limited to, procurement of substitute goods or
+services; loss of use, data, or profits; or business interruption)
+however caused and on any theory of liability, whether in contract,
+strict liability, or tort (including negligence or otherwise) arising in
+any way out of the use of this software, even if advised of the
+possibility of such damage.**

mps/design/guide.developer.txt

@@ -0,0 +1,100 @@
+.. mode: -*- rst -*-
+
+Review checklist
+================
+
+:Tag: guide.developer
+:Status: draft documentation
+:Author: Gareth Rees
+:Organization: Ravenbrook Limited
+:Date: 2018-09-18
+:Revision: $Id$
+:Copyright: See section `Copyright and License`_.
+:Index terms: pair: developer; guide
+
+
+Introduction
+------------
+
+_`.intro`: This is an introduction to the Memory Pool System (MPS) for
+new developers.
+
+
+What to read first
+------------------
+
+* `manual/build.txt`_ -- how to build the MPS.
+* `manual/guide`_ -- tutorial for the public interface.
+* `manual/topic`_ -- reference manual for the public interface.
+* `manual/code-index`_ -- description and purpose of each file of source code.
+* design.mps.config_ -- build configuration.
+* design.mps.tests_ -- how to run test cases.
+* design.mps.doc_ -- how to write and edit documentation.
+
+.. _manual/build.txt: https://www.ravenbrook.com/project/mps/master/manual/build.txt
+.. _manual/guide: https://www.ravenbrook.com/project/mps/master/manual/html/guide/index.html
+.. _manual/topic: https://www.ravenbrook.com/project/mps/master/manual/html/topic/index.html
+.. _manual/code-index: https://www.ravenbrook.com/project/mps/master/manual/html/code-index.html
+.. _design.mps.config: config
+.. _design.mps.tests: tests
+.. _design.mps.doc: doc
+
+
+References
+----------
+
+.. [APT_2018-09-17]
+   "Procedure for new developers";
+   Alistair Turnbull; Ravenbrook Limited; 2018-09-17;
+   <https://info.ravenbrook.com/mail/2018/09/17/12-36-45/0/>
+
+
+Document History
+----------------
+
+2018-09-18  GDR_  Created based on [APT_22018-09-17]_.
+
+.. _GDR: http://www.ravenbrook.com/consultants/gdr/
+
+
+Copyright and License
+---------------------
+
+Copyright © 2015-2016 Ravenbrook Limited <http://www.ravenbrook.com/>.
+All rights reserved. This is an open source license. Contact
+Ravenbrook for commercial licensing options.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+#. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+
+#. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+#. Redistributions in any form must be accompanied by information on how
+   to obtain complete source code for this software and any
+   accompanying software that uses this software.  The source code must
+   either be included in the distribution or be available for no more than
+   the cost of distribution plus a nominal fee, and must be freely
+   redistributable under reasonable conditions.  For an executable file,
+   complete source code means the source code for all modules it contains.
+   It does not include source code for modules or files that typically
+   accompany the major components of the operating system on which the
+   executable file runs.
+
+**This software is provided by the copyright holders and contributors
+"as is" and any express or implied warranties, including, but not
+limited to, the implied warranties of merchantability, fitness for a
+particular purpose, or non-infringement, are disclaimed.  In no event
+shall the copyright holders and contributors be liable for any direct,
+indirect, incidental, special, exemplary, or consequential damages
+(including, but not limited to, procurement of substitute goods or
+services; loss of use, data, or profits; or business interruption)
+however caused and on any theory of liability, whether in contract,
+strict liability, or tort (including negligence or otherwise) arising in
+any way out of the use of this software, even if advised of the
+possibility of such damage.**

mps/design/index.txt

@@ -54,11 +54,13 @@ collection_             Collection framework
 config_                 MPS configuration
 critical-path_          The critical path through the MPS
 diag_                   Diagnostic feedback
+doc_                    Documentation
 exec-env_               Execution environment
 failover_               Fail-over allocator
 finalize_               Finalization
 fix_                    The generic fix function
 freelist_               Free list allocator
+guide.developer_        Guide for new developers
 guide.hex.trans_        Transliterating the alphabet into hexadecimal
 guide.impl.c.format_    Coding standard: conventions for the general format of C source code in the MPS
 guide.impl.c.naming_    Coding standard: conventions for internal names
@@ -127,11 +129,13 @@ writef_                 The WriteF function
 .. _config: config
 .. _critical-path: critical-path
 .. _diag: diag
+.. _doc: doc
 .. _exec-env: exec-env
 .. _failover: failover
 .. _finalize: finalize
 .. _fix: fix
 .. _freelist: freelist
+.. _guide.developer: guide.developer
 .. _guide.hex.trans: guide.hex.trans
 .. _guide.impl.c.format: guide.impl.c.format
 .. _guide.impl.c.naming: guide.impl.c.naming
@@ -227,7 +231,7 @@ Document History
 Copyright and License
 ---------------------
 
-Copyright © 2002-2016 Ravenbrook Limited <http://www.ravenbrook.com/>.
+Copyright © 2002-2018 Ravenbrook Limited <http://www.ravenbrook.com/>.
 All rights reserved. This is an open source license. Contact
 Ravenbrook for commercial licensing options.
 

mps/design/tests.txt

@@ -21,27 +21,100 @@ tests.
 _`.readership`: This document is intended for any MPS developer.
 
 
-Smoke tests
+Running tests
------------
+-------------
 
-_`.smoke`: The "smoke tests" provide quick checks that the MPS is
+_`.run`: Run these commands::
-working. They run quickly enough for it to be practical to run them
+
-every time the MPS is built.
+    cd code
+    make -f <makefile> VARIETY=<variety> <target>  # Unix
+    nmake /f <makefile> VARIETY=<variety> <target> # Windows
+
+where ``<makefile>`` is the appropriate makefile for the platform (see
+manual/build.txt_), ``<variety>`` is the variety (see
+design.mps.config.var_) and ``<target>`` is the collection of tests
+(see `.target`_ below). For example::
+
+    make -f lii6ll VARIETY=cool testrun
+
+If ``<variety>`` is omitted, tests are run in both the cool and hot
+varieties.
+
+.. _design.mps.config.var: config#var
+.. _manual/build.txt: https://www.ravenbrook.com/project/mps/master/manual/build.txt
+
+
+Test targets
+------------
+
+_`.target`: The makefiles provide the following targets for common
+sets of tests:
+
+_`.target.testall`: The ``testall`` target runs all test cases (even
+if known to fail).
+
+_`.target.testrun`: The ``testrun`` target runs the "smoke tests".
+This subset of tests are quick checks that the MPS is working. They
+run quickly enough for it to be practical to run them every time the
+MPS is built.
+
+_`.target.testci`: The ``testci`` target runs the continuous
+integration tests (all tests that are known good according to the test database).
+
+_`.target.testansi`: The ``testansi`` target runs the subset of the
+smoke tests that can pass in the generic ("ANSI") build configuration
+(see design.mps.config.opt.ansi_).
+
+_`.target.testpollnone`: The ``testpollnone`` target runs the subset
+of the smoke tests that can pass in the generic ("ANSI") build
+configuration (see design.mps.config.opt.ansi_) with the option
+``CONFIG_POLL_NONE`` (see design.mps.config.opt.poll_).
+
+_`.target.testratio`: The ``testratio`` target compares the
+performance of the HOT and RASH varieties. See `.ratio`_.
+
+.. _design.mps.config.opt.ansi: config#opt.ansi
+.. _design.mps.config.opt.poll: config#opt.poll
+
+
+Test features
+-------------
 
 _`.randomize`: Each time a test case is run, it randomly chooses some
 of its parameters (for example, the sizes of objects, or how many
 links to create in a graph of references). This allows a fast test
 to cover many cases over time.
 
-_`.randomize.repeatable`: The random numbers are chosen
+_`.randomize.seed`: The random numbers are chosen
 pseudo-randomly based on a seed initialized from environmental data
 (the time and the processor cycle count). The seed is reported at test
-startup. Each test can be run with a specified seed. This ensures
+startup, for example::
-that the deterministic tests are repeatable.
+
+    $ xci6ll/cool/apss
+    xci6ll/cool/apss: randomize(): choosing initial state (v3): 2116709187.
+    ...
+    xci6ll/cool/apss: Conclusion: Failed to find any defects.
+
+Here, the number 2116709187 is the random seed.
+
+_`.randomize.specific-seed` Each test can be run with a specified seed
+by passing the seed on the command line, for example::
+
+    code$ xci6ll/cool/apss 2116709187
+    xci6ll/cool/apss: randomize(): resetting initial state (v3) to: 2116709187.
+    ...
+    xci6ll/cool/apss: Conclusion: Failed to find any defects.
+
+_`.randomize.repeatable`: This ensures that the deterministic tests
+are repeatable.
 
 
-Smoke test list
+Test list
-...............
+---------
+
+See `manual/code-index`_ for the full list of automated test cases.
+
+.. _manual/code-index: https://www.ravenbrook.com/project/mps/master/manual/html/code-index.html
 
 _`.test.finalcv`: Registers objects for finalization, makes them
 unreachable, deregisters them, etc. Churns to provoke minor (nursery)
@@ -56,10 +129,53 @@ _`.test.zcoll`: Collection scheduling, and collection feedback.
 _`.test.zmess`: Message lifecycle and finalization messages.
 
 
+Test database
+-------------
+
+_`.db`: The automated tests are described in the test database
+(tool/testcases.txt).
+
+_`.db.format`: This is a self-documenting plain-text database which
+gives for each test case its name and an optional set of features. For
+example the feature ``P`` means that the test case requires polling to
+succeed, and therefore is expected to fail in build configurations
+without polling (see design.mps.config.opt.poll_).
+
+_`.db.format.simple`: The format must be very simple because the test
+runner on Windows is written as a batch file (.bat), in order to avoid
+having to depend on any tools that are not standard with the operating
+system, and batch files are inflexible.
+
+_`.db.testrun`: The test runner (tool/testrun.sh on Unix or
+tool/testrun.bat on Windows) parses the test database to work out
+which tests to run according to the target. For example the
+``testpollnone`` target must skip all test cases with the ``P``
+feature.
+
+
+Test runner
+-----------
+
+_`.runner.req.automated`: The test runner must execute without user
+interaction, so that it can be used for continuous integration.
+
+_`.runner.req.output.pass`: Test cases are expected to pass nearly all the
+time, and in these cases we almost never want to see the output, so
+the test runner must suppress the output for passing tests.
+
+_`.runner.req.output.fail`: However, if a test case fails then the
+test runner must preserve the output from the failing test, including
+the random seed (see `.randomize.seed`_), so that this can be analyzed
+and the test repeated. Moreover, it must print the output from the
+failing test, so that if the test is being run on a continuous
+integration system like Travis, then the output of the failing tests
+is included in the failure report.
+
+
 Performance test
 ----------------
 
-_`.test.ratio`: The ``testratio`` target checks that the hot variety
+_`.ratio`: The ``testratio`` target checks that the hot variety
 is not too much slower than the rash variety. A failure of this test
 usually is expected to indicate that there are assertions on the
 critical path using ``AVER`` instead of ``AVER_CRITICAL`` (and so on).
@@ -68,12 +184,13 @@ the MVFF pool class, in the hot variety and the rash variety,
 computing the ratio of CPU time taken in the two varieties, and
 testing that this falls under an acceptable limit.
 
-Note that we don't use the elapsed time (as reported by the benchmark)
+_`.ratio.cpu-time`: Note that we use the CPU time (reported by
-because we want to be able to run this test on continuous integration
+``/usr/bin/time``) and not the elapsed time (as reported by the
-machines that might be heavily loaded.
+benchmark) because we want to be able to run this test on continuous
+integration machines that might be heavily loaded.
 
-This target is currently supported only on Unix platforms using GNU
+_`.ratio.platform`: This target is currently supported only on Unix
-Makefiles.
+platforms using GNU Makefiles.
 
 
 Adding a new smoke test
@@ -141,6 +258,54 @@ _`.new.manual`: Edit manual/source/code-index.rst and add the new test
 case to the "Automated test cases" section.
 
 
+Continuous integration
+----------------------
+
+_`.ci`: The MPS uses the following systems for continuous integration:
+
+_`.ci.travis`: The commercial Travis_ continuous integration service
+runs ``./configure && make install && make test`` (which exercises the
+``testci`` target for platforms lii6gc, lii6ll, xci6ll, and
+``testansi`` and ``testpollnone`` for platforms anangc and ananll)
+whenever there is a commit to the `mps GitHub repository`_. See the
+`Ravenbrook/mps`_ project on Travis for a build history.
+
+.. _mps GitHub repository: https://github.com/ravenbrook/mps
+.. _Travis: https://travis-ci.org
+.. _Ravenbrook/mps: https://travis-ci.org/Ravenbrook/mps
+
+_`.ci.travis.config`: Travis is configured using the .travis.yml file
+at top level of the project.
+
+_`.ci.jenkins`: An instance of Jenkins_ running on "gannet" (a Windows
+PC at the Ravenbrook office) runs the ``testci`` target on platforms
+w3i3mv and w3i6mv whenever there is a commit to the `mps GitHub
+repository`_.
+
+_`.ci.jenkins.config`: There are instructions for installing and
+configuring Jenkins in [GDR_2016-04-14]_, [GDR_2016-04-15]_ and
+[GDR_2016-04-20]_.
+
+
+References
+----------
+
+.. [GDR_2016-04-14]
+   "Jenkins setup and configuration procedure";
+   Gareth Rees; Ravenbrook Limited; 2016-04-14;
+   <https://info.ravenbrook.com/mail/2016/04/14/17-19-08/0/>.
+
+.. [GDR_2016-04-15]
+   "Re: Jenkins setup and configuration procedure";
+   Gareth Rees; Ravenbrook Limited; 2016-04-15;
+   <https://info.ravenbrook.com/mail/2016/04/15/09-03-53/0/>.
+
+.. [GDR_2016-04-20]
+   "Re: Jenkins setup and configuration procedure";
+   Gareth Rees; Ravenbrook Limited; 2016-04-20;
+   <https://info.ravenbrook.com/mail/2016/04/20/13-26-33/0/>.
+
+
 Document History
 ----------------
 

mps/manual/source/design/index.rst

@@ -13,10 +13,12 @@ Design
     clock
     config
     critical-path
+    doc
     exec-env
     failover
     finalize
     freelist
+    guide.developer
     guide.hex.trans
     guide.impl.c.format
     guide.impl.c.naming