This repository contains tools for managing large-scale QA tests, such as the archive rebuilds performed on AWS.

The overall process (detailed below) looks like:

1. create a set of nodes to use as build nodes
2. generate a list of tasks
3. start the process, using cqa-masternode
4. wait
5. fetch the results, using cqa-fetch-results
6. process the results (file bugs), using cqa-scanlogs and cqa-annotate

Note: all those tools are not really release-quality. Most notably, many things are hardcoded.

Create a set of nodes to use as build nodes

This is the part that is Cloud-provider-specific. For AWS EC2, use cqa-ec2. Adjust if another provider is in use.

Read/change the top of bin/cqa-ec2 where all parameters are defined.

Look at bin/cqa-setup-rebuildnode (script that is run on the nodes after it has been provisioned by the provider, to set it up as a build node)

Run everything on qa2-master: that machine is allowed access to the EC2 API to create nodes.

bin/cqa-ec2 help

only one node:

bin/cqa-ec2 create_and_setup

or with a specific number of nodes:

bin/cqa-ec2 create_and_setup --nb=50

check that VMs have been started with bin/cqa-ec2 list-all

Create a list of nodes:

bin/cqa-ec2 list > nodes

Generate the list of tasks

Tasks can be anything supported by cqa-process-task.

See examples in lib/collab-qa/tasks_generator.rb.

One example:

ruby -Ilib -rcollab-qa -e "
tasks = CollabQA::TasksGenerator::new('rebuild-full').generate({'exclude-not-testing' => true})
tasks = tasks.select { |t| t['esttime'] && t['esttime'] < 30 }.shuffle[0...100]
puts JSON::pretty_generate(tasks)
" > tasks.json

Start the process and wait

cqa-masternode is the orchestrator that decides which build(s) to start on each node. See cqa-masternode -h

Example invocation: bin/cqa-masternode -n nodes -t tasks.json -s 3 -o log -d /tmp/logs.test --no-halt

This will:

• build tasks in tasks.json, using nodes in nodes
• first build 3 tasks per node (-s 3), then retry failed builds alone on a node (because --no-retry is not specified)
• not halt the nodes when they are no longer useful (because --no-halt)

To watch progress, use the log file, the output directory (-d), and bin/cqa-ec2 list-all (if AWS EC2) to see nodes that are still running

Fetch the results

• make sure that all instances are terminated (sometimes they stay running for random reasons) using bin/cqa-ec2 list-all
  • Don't terminate instances not involved in your rebuild!
• cd to collab-qa-data directory
• run (for example): DATE=2018/12/29 bin/cqa-fetch-results -d /tmp/logs.test
• then follow the instructions

Process results

• Use cqa-scanlogs and cqa-annotate
  • Generate a list of failures: cqa-scanlogs |grep -v ' OK ' > /tmp/list
  • Generate a TODOFILE: cqa-merge-results /dev/null /tmp/list > /tmp/todofile
  • Edit lib/collab-qa/log-parser.rb to add a template.
  • Set the BUGTYPE environment variable to your template.
  • Use cqa-fetchbugs -t /tmp/todofile to fetch candidate bugs from the BTS
  • Use cqa-annotate -t /tmp/todofile -r regexp to start filing bugs
    • Enter 's' to skip a package
    • Enter 'i' to ignore this package permanently
    • Enter a bug number (1,2,3,..) to annotate the TODOFILE with that bug
    • Enter 'r' to report a new bug
    • Enter 'f' to view the full log file on PAGER

Random notes

• Get reverse build-dependencies of a package: apt-get install -y --no-install-recommends grep-dctrl devscripts dose-extra

for p in $(grep-aptavail -S boost-defaults -sPackage -n); do build-rdeps -q $p > build-rdeps.$p; done

build-rdeps -q $(grep-aptavail -S boost-defaults -sPackage -n) > build-rdeps

• Extract sources from tasks: jq -r '. | map(.source) | .[]' < tasks.json