GitLab

Originally created by @anonym on #9491 (Redmine)

Now that #9424 (closed) is solved, we could do something pretty interesting. We could remove the --debug option and replace it with a formatter that includes debug information. It’d be nice to call it simply debug, but it already exists (and is useful when developing new formatters, apparently), but we could override it with our one, or pick a new name, like pretty_debug. So, this formatter would be the pretty one, but also have all the debug info we print to STDERR, and the info we let some subprocesses (sikuli, guestfs) print to STDERR when --debug is enabled.

To make a formatter that we can print to with e.g. a debug_log() function is pretty straight-forward:

def debug_log(message)
  return if not($config["DEBUG"])
  $debug_log_fns.each { |fn| fn.call(message) }
end

require 'cucumber/formatter/pretty'
module ExtraFormatters
  class PrettyDebug < Cucumber::Formatter::Pretty
    def initialize(*args)
      super(*args)
      $debug_log_fns ||= []
      $debug_log_fns << self.method(:debug_log)
    end

    def debug_log(msg)
      @io.puts(msg)
    end
  end
end

module Cucumber
  module Cli
    class Options
      BUILTIN_FORMATS['debug'] = ['ExtraFormatters::PrettyDebug', 'Prints the feature with debuggin information - in colours.']
    end
  end
end

The hard part is to connect the STDOUT/STDERR of subprocesses so it writes to the formatters destination (which can be a file or the terminal/stdout). I only have some vague ideas based on crazy stuff that the JUnit formatter does with some sort of stream interceptor. Not sure at all if that will work out though.

Now, why would we want to do this? Well, the output when --debug is enabled is quite hard to follow given the amount of debugging info we print (especially for Tor leak tests) so I generally run without it. However, when there’s an issue I often regret not having the debug info, since some issues do not reappear every time due to our robustness issues. But if we had this we could run cucumber with --format pretty --format debug --out /tmp/debug.log so that we get the pretty non-debug written to the terminal that is easy to follow, and the full debug info (including the pretty output, for the all-important context) written to a file that we can investigate if there’s some unexpected error. Awesome! Currently we can do the opposite (pretty + debug to the terminal, pretty only to file) with --debug -- --format pretty --format pretty --out /tmp/pretty.log which isn’t as nice, but I actually do this quite often.

Note if we have this we can also kill run_test_suite’s --log-to-file option.

Feature Branch: test/9491-debug-cucumber-formatter