Index of /openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-rocky/bddeedd/logs

Name	Last modified	Size

Parent Directory		-
_No_failure_reason_found.log	2020-03-28 06:32	40
ara.json	2020-03-28 06:32	0
ara.oooq.oc.json	2020-03-28 06:32	0
ara.oooq.uc.json	2020-03-28 06:32	0
dns_cache.txt	2020-03-28 06:23	21K
full_logs.html	2020-03-28 06:33	530
quickstart_collect_logs.log	2020-03-28 06:33	5.1M
quickstart_files/	2020-04-29 06:33	-
quickstart_install.log	2020-03-28 06:23	96K
reproducer-quickstart-deprecated/	2020-03-28 06:32	-
reproducer-quickstart/	2020-03-28 06:33	-
subnode-1/	2020-03-28 06:31	-
testrepository.subunit.gz	2020-03-28 06:33	68
undercloud/	2020-03-28 06:31	-
zuul_console.json	2020-03-28 06:23	2.4M

Name

Last modified

Size

Parent Directory

_No_failure_reason_found.log

2020-03-28 06:32

2020-03-28 06:32

2020-03-28 06:32

2020-03-28 06:32

2020-03-28 06:23

21K

full_logs.html

2020-03-28 06:33

530

quickstart_collect_logs.log

2020-03-28 06:33

5.1M

quickstart_files/

2020-04-29 06:33

quickstart_install.log

2020-03-28 06:23

96K

reproducer-quickstart-deprecated/

2020-03-28 06:32

reproducer-quickstart/

2020-03-28 06:33

subnode-1/

2020-03-28 06:31

testrepository.subunit.gz

2020-03-28 06:33

undercloud/

2020-03-28 06:31

zuul_console.json

2020-03-28 06:23

2.4M

README for Quickstart Logs

Links to important log files

undercloud/var/log/extra/errors.txt.txt.gz - the concatenation of all the errors on any node in a single file

undercloud/home/zuul/ - the source and log output of all templated shell scripts

undercloud/var/log/ - directories and files copied from /var/log on the undercloud. If other overcloud/subnodes exist, similar $node/var/log directories will also exist in these logs.

undercloud/var/log/extra/ - extra system details like package list, and cpu info gathered from the undercloud

undercloud/var/log/extra/rpm-list.txt.gz - rpms installed to the undercloud, container rpms can be found in extra//containers/$container/info.log

undercloud/var/lib/mistral - output of all ansible used by config-download to drive the overcloud deployment

undercloud/etc/yum.repos.d/ - link to the yum.repos.d directory

undercloud/etc/yum.repos.d/delorean.repo - link to the delorean.repo ( the build )

Links container log files

Links to package builds and tempest

delorean_logs - the source code change built into rpm and the build logs are here

undercloud/var/log/tempest/ - Tempest run results logs

undercloud/home/zuul/tempest/etc/ - tempest.conf and tempest blacklist and whitelist tests

stackviz - stackviz tempest test results

Links to puppet logs

undercloud/var/lib/config-data/ - the configuration files for each openstack service

undercloud/etc/puppet/hieradata/ - the hieradata of the deployment, service-configs, net and vip info

undercloud/var/lib/container-puppet - puppet config for each container

Check the base directory for a file called *failure_reason* for automatic failure detection. If no known error has been found the file will be named "No_failure_reason_found".

Most of the undercloud and overcloud deployment log files can be found in undercloud/home/zuul tracebacks and other errors are collected in the following log per node:

undercloud/var/log/extra/errors.txt.txt.gz - the concatenation of all the errors on any node in a single file

Next check the console log and search for PLAY RECAP. There are sometimes multiple ansible runs in a job, usually the last one is the relevant.
If no PLAY RECAP text is found that usually means an infra failure before Quickstart could even start.
If this is a different Ansible error, that could mean either an infra problem (often has UNREACHABLE in the line) or a bug in Quickstart.

Ask on #tripleo to get help or open a bug on Launchpad. Add the "ci" tag if it's a CI issue and "quickstart" if you suspect that the bug is in Quickstart itself.

Finally try rechecking or asking on #tripleo

The logs contain files showing variables used in the job run.

undercloud/var/log/extra/dump_variables_vars.json.txt.gz - contains the variables used in the running the actual test
releases.sh - the output of the script setting release-related variables
playbook_executions.log - prints out the complete commands, with all expanded arguments, to run the playbooks

Upstream OpenStack Health, Elastic Search and Kibana

http://status.openstack.org/elastic-recheck/ - A tool to track the impact of known bugs in OpenStack CI
http://logstash.openstack.org - filter in details in the log files from OpenStack CI
OpenStack Health - upstream job results by project

Tools that will help you spot a trend in TripleO CI

Job Exploration - check the job history across upstream clouds
zuul job filter - zuul's job filter per job
http://cistatus.tripleo.org - Overall check job status
http://cistatus.tripleo.org/gates/ - Overall gate job status

Tools to compare one job to another.

jcomparison - A tool to compare results from one job to another
log reduce - A tool that uses AI features to reduce the noise in logs and present only what is needed for debug

As a debugging step, a job can be run manually with '-dryrun' appended to the job name. When the "playbook dry run" option is invoked, the playbooks will not execute and collect logs will not run but certain log files, including 'toci_env_args_output.log', which contains the environment variables used in the job, and playbook_executions.log will still be produced in the logs directory for inspection. This option serves to assist with debugging and to test the testing scripts themselves.