From f6721224bf9532ffc8ab85c9567da67b2209bfea Mon Sep 17 00:00:00 2001 From: Gareth Rees Date: Tue, 18 Sep 2018 12:50:29 +0100 Subject: [PATCH] Guide for new developers. Documentation design. Expand test case design. Copied from Perforce Change: 195083 --- mps/code/comm.gmk | 10 +- mps/design/config.txt | 6 +- mps/design/doc.txt | 149 ++++++++++++++++++++++ mps/design/guide.developer.txt | 100 +++++++++++++++ mps/design/index.txt | 6 +- mps/design/tests.txt | 197 ++++++++++++++++++++++++++--- mps/manual/source/design/index.rst | 2 + 7 files changed, 442 insertions(+), 28 deletions(-) create mode 100644 mps/design/doc.txt create mode 100644 mps/design/guide.developer.txt diff --git a/mps/code/comm.gmk b/mps/code/comm.gmk index d323b3ea67d..30b20d7f755 100644 --- a/mps/code/comm.gmk +++ b/mps/code/comm.gmk @@ -3,7 +3,7 @@ # comm.gmk: COMMON GNUMAKEFILE FRAGMENT # # $Id$ -# Copyright (c) 2001-2016 Ravenbrook Limited. See end of file for license. +# Copyright (c) 2001-2018 Ravenbrook Limited. See end of file for license. # # DESCRIPTION # @@ -308,11 +308,7 @@ all: $(ALL_TARGETS) # == Automated test suites == # -# testrun = "smoke test", fast enough to run before every commit -# testci = continuous integration tests, must be known good -# testall = all test cases, for ensuring quality of a release -# testansi = tests that run on the generic ("ANSI") platform -# testpollnone = tests that run on the generic platform with CONFIG_POLL_NONE +# See design.mps.tests.target. TEST_SUITES=testrun testci testall testansi testpollnone @@ -714,7 +710,7 @@ find-puns: phony # C. COPYRIGHT AND LICENSE # -# Copyright (c) 2001-2016 Ravenbrook Limited . +# Copyright (c) 2001-2018 Ravenbrook Limited . # All rights reserved. This is an open source license. Contact # Ravenbrook for commercial licensing options. # diff --git a/mps/design/config.txt b/mps/design/config.txt index 3bf1d96e29c..903ce418236 100644 --- a/mps/design/config.txt +++ b/mps/design/config.txt @@ -164,9 +164,7 @@ predefined when compiling the module sources:: CONFIG_VAR_ -The variety-codes are the letter code that appears after "variety." in -the tag of the relevant variety document (see variety.*) converted to -upper case. Currently (2012-09-03): +_`.var`: The variety codes are as follows: _`.var.hot`: ``HOT`` @@ -621,7 +619,7 @@ Document History Copyright and License --------------------- -Copyright © 2013-2016 Ravenbrook Limited . +Copyright © 2013-2018 Ravenbrook Limited . All rights reserved. This is an open source license. Contact Ravenbrook for commercial licensing options. diff --git a/mps/design/doc.txt b/mps/design/doc.txt new file mode 100644 index 00000000000..84c88308517 --- /dev/null +++ b/mps/design/doc.txt @@ -0,0 +1,149 @@ +.. mode: -*- rst -*- + +Documentation +============= + +:Tag: design.mps.doc +:Author: Gareth Rees +:Date: 2018-09-18 +:Status: draft design +:Revision: $Id$ +:Copyright: See section `Copyright and License`_. +:Index terms: pair: documentation; design + + +Introduction +------------ + +_`.intro`: This is the design of the documentation system for the +Memory Pool System. + +_`.readership`: This document is intended for any MPS developer. + + +Types +----- + +_`.type`: The MPS has multiple types of documentation, suitable for +different audiences. + +_`.type.comment`: Comments in the code provide information that is +required in order for developers to make correct edits to nearby code. +(Audience: MPS developers editing nearby code.) + +_`.type.design`: Design documentation lists requirements and explains +how the code meets the requirements. (Audience: MPS developers working +on a subsystem.) + +_`.type.devguide`: Developer guides provide general guidance for +developers, not specific to any particular subsystem. (Audience: MPS +developers generally.) + +_`.type.procedure`: Procedures list the steps for carrying out +development tasks. (Audience: MPS developers who need to carry out +particular tasks reliably.) + +_`.type.tutorial`: Tutorials describe how to use the MPS to meet +client program requirements. (Audience: beginner client program +developers.) + +_`.type.reference`: Reference documentation specifies the public +features of the MPS. (Audience: expert client program developers.) + +_`.type.mmref`: The Memory Management Reference describes general +principles of memory management, with cross-references to the MPS +documentation. (Audience: the world.) + + +Requirements +------------ + +_`.req.source`: Derived from [RB_2013-05-09]_. + +_`.req.easy`: It must be easy to read and write documentation must +using standard text editors. Barriers to documentation must be low. + +_`.req.presentation`: It must be possible to process documentation +into presentation formats, for example web pages. + +_`.req.single-source`: Documents must have a single source. Processing +into other formats must be automatic and not depend on hand editing or +maintaining parallel versions. + +_`.req.durable`: The format of documents should be supported for the +foreseeable future. It must not require continual updating to keep up +with changes to processing software. + +_`.req.design.ref`: It must be easy to reference points made in design +documents from the code. + +_`.req.design.standalone`: Design documents must stand alone: they +must not require particular software to make them readable or +complete. + + +Implementation +-------------- + +TODO + + + +References +---------- + +.. [RB_2013-05-09] + "MPS design document format and process"; + Richard Brooksby; Ravenbrook Limited; 2013-05-09; + . + + +Document History +---------------- + +- 2018-09-18 GDR_ Created based on [RB_2013-05-09]_. + +.. _GDR: http://www.ravenbrook.com/consultants/gdr/ + + +Copyright and License +--------------------- + +Copyright © 2018 Ravenbrook Limited . +All rights reserved. This is an open source license. Contact +Ravenbrook for commercial licensing options. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +#. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +#. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +#. Redistributions in any form must be accompanied by information on how + to obtain complete source code for this software and any + accompanying software that uses this software. The source code must + either be included in the distribution or be available for no more than + the cost of distribution plus a nominal fee, and must be freely + redistributable under reasonable conditions. For an executable file, + complete source code means the source code for all modules it contains. + It does not include source code for modules or files that typically + accompany the major components of the operating system on which the + executable file runs. + +**This software is provided by the copyright holders and contributors +"as is" and any express or implied warranties, including, but not +limited to, the implied warranties of merchantability, fitness for a +particular purpose, or non-infringement, are disclaimed. In no event +shall the copyright holders and contributors be liable for any direct, +indirect, incidental, special, exemplary, or consequential damages +(including, but not limited to, procurement of substitute goods or +services; loss of use, data, or profits; or business interruption) +however caused and on any theory of liability, whether in contract, +strict liability, or tort (including negligence or otherwise) arising in +any way out of the use of this software, even if advised of the +possibility of such damage.** diff --git a/mps/design/guide.developer.txt b/mps/design/guide.developer.txt new file mode 100644 index 00000000000..ed854b4f7e6 --- /dev/null +++ b/mps/design/guide.developer.txt @@ -0,0 +1,100 @@ +.. mode: -*- rst -*- + +Review checklist +================ + +:Tag: guide.developer +:Status: draft documentation +:Author: Gareth Rees +:Organization: Ravenbrook Limited +:Date: 2018-09-18 +:Revision: $Id$ +:Copyright: See section `Copyright and License`_. +:Index terms: pair: developer; guide + + +Introduction +------------ + +_`.intro`: This is an introduction to the Memory Pool System (MPS) for +new developers. + + +What to read first +------------------ + +* `manual/build.txt`_ -- how to build the MPS. +* `manual/guide`_ -- tutorial for the public interface. +* `manual/topic`_ -- reference manual for the public interface. +* `manual/code-index`_ -- description and purpose of each file of source code. +* design.mps.config_ -- build configuration. +* design.mps.tests_ -- how to run test cases. +* design.mps.doc_ -- how to write and edit documentation. + +.. _manual/build.txt: https://www.ravenbrook.com/project/mps/master/manual/build.txt +.. _manual/guide: https://www.ravenbrook.com/project/mps/master/manual/html/guide/index.html +.. _manual/topic: https://www.ravenbrook.com/project/mps/master/manual/html/topic/index.html +.. _manual/code-index: https://www.ravenbrook.com/project/mps/master/manual/html/code-index.html +.. _design.mps.config: config +.. _design.mps.tests: tests +.. _design.mps.doc: doc + + +References +---------- + +.. [APT_2018-09-17] + "Procedure for new developers"; + Alistair Turnbull; Ravenbrook Limited; 2018-09-17; + + + +Document History +---------------- + +2018-09-18 GDR_ Created based on [APT_22018-09-17]_. + +.. _GDR: http://www.ravenbrook.com/consultants/gdr/ + + +Copyright and License +--------------------- + +Copyright © 2015-2016 Ravenbrook Limited . +All rights reserved. This is an open source license. Contact +Ravenbrook for commercial licensing options. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +#. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +#. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +#. Redistributions in any form must be accompanied by information on how + to obtain complete source code for this software and any + accompanying software that uses this software. The source code must + either be included in the distribution or be available for no more than + the cost of distribution plus a nominal fee, and must be freely + redistributable under reasonable conditions. For an executable file, + complete source code means the source code for all modules it contains. + It does not include source code for modules or files that typically + accompany the major components of the operating system on which the + executable file runs. + +**This software is provided by the copyright holders and contributors +"as is" and any express or implied warranties, including, but not +limited to, the implied warranties of merchantability, fitness for a +particular purpose, or non-infringement, are disclaimed. In no event +shall the copyright holders and contributors be liable for any direct, +indirect, incidental, special, exemplary, or consequential damages +(including, but not limited to, procurement of substitute goods or +services; loss of use, data, or profits; or business interruption) +however caused and on any theory of liability, whether in contract, +strict liability, or tort (including negligence or otherwise) arising in +any way out of the use of this software, even if advised of the +possibility of such damage.** diff --git a/mps/design/index.txt b/mps/design/index.txt index 2712128a2cc..08d4d23ab96 100644 --- a/mps/design/index.txt +++ b/mps/design/index.txt @@ -54,11 +54,13 @@ collection_ Collection framework config_ MPS configuration critical-path_ The critical path through the MPS diag_ Diagnostic feedback +doc_ Documentation exec-env_ Execution environment failover_ Fail-over allocator finalize_ Finalization fix_ The generic fix function freelist_ Free list allocator +guide.developer_ Guide for new developers guide.hex.trans_ Transliterating the alphabet into hexadecimal guide.impl.c.format_ Coding standard: conventions for the general format of C source code in the MPS guide.impl.c.naming_ Coding standard: conventions for internal names @@ -127,11 +129,13 @@ writef_ The WriteF function .. _config: config .. _critical-path: critical-path .. _diag: diag +.. _doc: doc .. _exec-env: exec-env .. _failover: failover .. _finalize: finalize .. _fix: fix .. _freelist: freelist +.. _guide.developer: guide.developer .. _guide.hex.trans: guide.hex.trans .. _guide.impl.c.format: guide.impl.c.format .. _guide.impl.c.naming: guide.impl.c.naming @@ -227,7 +231,7 @@ Document History Copyright and License --------------------- -Copyright © 2002-2016 Ravenbrook Limited . +Copyright © 2002-2018 Ravenbrook Limited . All rights reserved. This is an open source license. Contact Ravenbrook for commercial licensing options. diff --git a/mps/design/tests.txt b/mps/design/tests.txt index 47270217866..e8994d94d9f 100644 --- a/mps/design/tests.txt +++ b/mps/design/tests.txt @@ -21,27 +21,100 @@ tests. _`.readership`: This document is intended for any MPS developer. -Smoke tests ------------ +Running tests +------------- -_`.smoke`: The "smoke tests" provide quick checks that the MPS is -working. They run quickly enough for it to be practical to run them -every time the MPS is built. +_`.run`: Run these commands:: + + cd code + make -f VARIETY= # Unix + nmake /f VARIETY= # Windows + +where ```` is the appropriate makefile for the platform (see +manual/build.txt_), ```` is the variety (see +design.mps.config.var_) and ```` is the collection of tests +(see `.target`_ below). For example:: + + make -f lii6ll VARIETY=cool testrun + +If ```` is omitted, tests are run in both the cool and hot +varieties. + +.. _design.mps.config.var: config#var +.. _manual/build.txt: https://www.ravenbrook.com/project/mps/master/manual/build.txt + + +Test targets +------------ + +_`.target`: The makefiles provide the following targets for common +sets of tests: + +_`.target.testall`: The ``testall`` target runs all test cases (even +if known to fail). + +_`.target.testrun`: The ``testrun`` target runs the "smoke tests". +This subset of tests are quick checks that the MPS is working. They +run quickly enough for it to be practical to run them every time the +MPS is built. + +_`.target.testci`: The ``testci`` target runs the continuous +integration tests (all tests that are known good according to the test database). + +_`.target.testansi`: The ``testansi`` target runs the subset of the +smoke tests that can pass in the generic ("ANSI") build configuration +(see design.mps.config.opt.ansi_). + +_`.target.testpollnone`: The ``testpollnone`` target runs the subset +of the smoke tests that can pass in the generic ("ANSI") build +configuration (see design.mps.config.opt.ansi_) with the option +``CONFIG_POLL_NONE`` (see design.mps.config.opt.poll_). + +_`.target.testratio`: The ``testratio`` target compares the +performance of the HOT and RASH varieties. See `.ratio`_. + +.. _design.mps.config.opt.ansi: config#opt.ansi +.. _design.mps.config.opt.poll: config#opt.poll + + +Test features +------------- _`.randomize`: Each time a test case is run, it randomly chooses some of its parameters (for example, the sizes of objects, or how many links to create in a graph of references). This allows a fast test to cover many cases over time. -_`.randomize.repeatable`: The random numbers are chosen +_`.randomize.seed`: The random numbers are chosen pseudo-randomly based on a seed initialized from environmental data (the time and the processor cycle count). The seed is reported at test -startup. Each test can be run with a specified seed. This ensures -that the deterministic tests are repeatable. +startup, for example:: + + $ xci6ll/cool/apss + xci6ll/cool/apss: randomize(): choosing initial state (v3): 2116709187. + ... + xci6ll/cool/apss: Conclusion: Failed to find any defects. + +Here, the number 2116709187 is the random seed. + +_`.randomize.specific-seed` Each test can be run with a specified seed +by passing the seed on the command line, for example:: + + code$ xci6ll/cool/apss 2116709187 + xci6ll/cool/apss: randomize(): resetting initial state (v3) to: 2116709187. + ... + xci6ll/cool/apss: Conclusion: Failed to find any defects. + +_`.randomize.repeatable`: This ensures that the deterministic tests +are repeatable. -Smoke test list -............... +Test list +--------- + +See `manual/code-index`_ for the full list of automated test cases. + +.. _manual/code-index: https://www.ravenbrook.com/project/mps/master/manual/html/code-index.html _`.test.finalcv`: Registers objects for finalization, makes them unreachable, deregisters them, etc. Churns to provoke minor (nursery) @@ -56,10 +129,53 @@ _`.test.zcoll`: Collection scheduling, and collection feedback. _`.test.zmess`: Message lifecycle and finalization messages. +Test database +------------- + +_`.db`: The automated tests are described in the test database +(tool/testcases.txt). + +_`.db.format`: This is a self-documenting plain-text database which +gives for each test case its name and an optional set of features. For +example the feature ``P`` means that the test case requires polling to +succeed, and therefore is expected to fail in build configurations +without polling (see design.mps.config.opt.poll_). + +_`.db.format.simple`: The format must be very simple because the test +runner on Windows is written as a batch file (.bat), in order to avoid +having to depend on any tools that are not standard with the operating +system, and batch files are inflexible. + +_`.db.testrun`: The test runner (tool/testrun.sh on Unix or +tool/testrun.bat on Windows) parses the test database to work out +which tests to run according to the target. For example the +``testpollnone`` target must skip all test cases with the ``P`` +feature. + + +Test runner +----------- + +_`.runner.req.automated`: The test runner must execute without user +interaction, so that it can be used for continuous integration. + +_`.runner.req.output.pass`: Test cases are expected to pass nearly all the +time, and in these cases we almost never want to see the output, so +the test runner must suppress the output for passing tests. + +_`.runner.req.output.fail`: However, if a test case fails then the +test runner must preserve the output from the failing test, including +the random seed (see `.randomize.seed`_), so that this can be analyzed +and the test repeated. Moreover, it must print the output from the +failing test, so that if the test is being run on a continuous +integration system like Travis, then the output of the failing tests +is included in the failure report. + + Performance test ---------------- -_`.test.ratio`: The ``testratio`` target checks that the hot variety +_`.ratio`: The ``testratio`` target checks that the hot variety is not too much slower than the rash variety. A failure of this test usually is expected to indicate that there are assertions on the critical path using ``AVER`` instead of ``AVER_CRITICAL`` (and so on). @@ -68,12 +184,13 @@ the MVFF pool class, in the hot variety and the rash variety, computing the ratio of CPU time taken in the two varieties, and testing that this falls under an acceptable limit. -Note that we don't use the elapsed time (as reported by the benchmark) -because we want to be able to run this test on continuous integration -machines that might be heavily loaded. +_`.ratio.cpu-time`: Note that we use the CPU time (reported by +``/usr/bin/time``) and not the elapsed time (as reported by the +benchmark) because we want to be able to run this test on continuous +integration machines that might be heavily loaded. -This target is currently supported only on Unix platforms using GNU -Makefiles. +_`.ratio.platform`: This target is currently supported only on Unix +platforms using GNU Makefiles. Adding a new smoke test @@ -141,6 +258,54 @@ _`.new.manual`: Edit manual/source/code-index.rst and add the new test case to the "Automated test cases" section. +Continuous integration +---------------------- + +_`.ci`: The MPS uses the following systems for continuous integration: + +_`.ci.travis`: The commercial Travis_ continuous integration service +runs ``./configure && make install && make test`` (which exercises the +``testci`` target for platforms lii6gc, lii6ll, xci6ll, and +``testansi`` and ``testpollnone`` for platforms anangc and ananll) +whenever there is a commit to the `mps GitHub repository`_. See the +`Ravenbrook/mps`_ project on Travis for a build history. + +.. _mps GitHub repository: https://github.com/ravenbrook/mps +.. _Travis: https://travis-ci.org +.. _Ravenbrook/mps: https://travis-ci.org/Ravenbrook/mps + +_`.ci.travis.config`: Travis is configured using the .travis.yml file +at top level of the project. + +_`.ci.jenkins`: An instance of Jenkins_ running on "gannet" (a Windows +PC at the Ravenbrook office) runs the ``testci`` target on platforms +w3i3mv and w3i6mv whenever there is a commit to the `mps GitHub +repository`_. + +_`.ci.jenkins.config`: There are instructions for installing and +configuring Jenkins in [GDR_2016-04-14]_, [GDR_2016-04-15]_ and +[GDR_2016-04-20]_. + + +References +---------- + +.. [GDR_2016-04-14] + "Jenkins setup and configuration procedure"; + Gareth Rees; Ravenbrook Limited; 2016-04-14; + . + +.. [GDR_2016-04-15] + "Re: Jenkins setup and configuration procedure"; + Gareth Rees; Ravenbrook Limited; 2016-04-15; + . + +.. [GDR_2016-04-20] + "Re: Jenkins setup and configuration procedure"; + Gareth Rees; Ravenbrook Limited; 2016-04-20; + . + + Document History ---------------- diff --git a/mps/manual/source/design/index.rst b/mps/manual/source/design/index.rst index 5e75b6ec6d7..27dfd8c0f08 100644 --- a/mps/manual/source/design/index.rst +++ b/mps/manual/source/design/index.rst @@ -13,10 +13,12 @@ Design clock config critical-path + doc exec-env failover finalize freelist + guide.developer guide.hex.trans guide.impl.c.format guide.impl.c.naming