• Matthew Jordan's avatar
    testsuite: Improve Logging · 1f5f0e90
    Matthew Jordan authored
    This patch changes up the logging in the Asterisk Test Suite. It reduces the
    amount of logging done in the logs/ directory by default (although this can
    always be bumped up via logger.conf), and it redirects the vast majority of
    the test suite logs to the actual sandboxed test run directories themselves.
    
    When we first added Python logging, there were far fewer tests. Having all
    tests dump into the same log files actually felt like a benefit, since
    during test development (or when developing many tests) it made it easier to
    debug test failures. Over time, however, this approach has run into a number
    of problems:
    (1) Many of the tests are much 'chattier' than they used to be. Having twisted,
        starpy, websockets, requests, and other libraries dump information has
        greatly increased the number of log messages.
    (2) There are a lot more tests than there were when this was added.
    (3) Tailing a log file is not always the best way to debug. Sometimes you have
        to search through an entire log file for one run. Finding the error
        becomes problematic when your editor of choice chokes on the size of the
        file. This is particularly true for the continuous integration agents,
        which have to include incredibly large log files when a test fails.
    
    Hence, this patch.
    
    Logging now works in the following way:
    (1) The test_case.TestCase class now always sets up the logging. The logging
        set up done by test_runner was removed (as it only logged out a few
        messages before an instance of TestCase was created anyway).
    (2) A global logger can still be configured in logger.conf. It now only sets up
        a logger of INFO messages and higher. This allows a test executor to watch
        which tests are being run, without getting spammed. During test
        development, the log message level can be increased to DEBUG.
    (3) TestCase now places the Asterisk directories created by the test execution
        in a further subfolder, run_N (where N increases with each execution of
        that test). Where before you might have:
    
          tests/my_test/ast1
          tests/my_test/ast2
          tests/my_test/ast3
          tests/my_test/ast4
    
        You will now have:
    
          tests/my_test/run_1/ast1
          tests/my_test/run_1/ast2
          tests/my_test/run_2/ast1
          tests/my_test/run_2/ast2
    
        This lets you determine which Asterisk instances belong together, and also
        makes it possible for only the erroring test run to be archived when a test
        fails (as opposed to every Asterisk directory)
    (4) TestCase now creates a DEBUG and INFO log file(s) in the run directory.
        These contain the full Asterisk logs for the test run.
    
    Review: https://reviewboard.asterisk.org/r/3489/
    
    
    
    git-svn-id: http://svn.digium.com/svn/testsuite/asterisk/trunk@5029 f64d8a5a-fed3-4b5f-9ffa-d34064e55e19
    1f5f0e90