Index of /openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-stein/ba5c886/logs

Name	Last modified	Size

Parent Directory		-
quickstart_files/	2020-03-28 06:29	-
testrepository.subunit.gz	2020-03-28 06:29	68
quickstart_collect_logs.log	2020-03-28 06:29	4.8M
full_logs.html	2020-03-28 06:29	530
reproducer-quickstart/	2020-03-28 06:29	-
reproducer-quickstart-deprecated/	2020-03-28 06:29	-
ara.oooq.oc.json	2020-03-28 06:29	0
ara.oooq.uc.json	2020-03-28 06:29	0
ara.json	2020-03-28 06:29	0
_No_failure_reason_found.log	2020-03-28 06:29	40
undercloud/	2020-03-28 06:28	-
subnode-1/	2020-03-28 06:27	-
dns_cache.txt	2020-03-28 06:20	19K
zuul_console.json	2020-03-28 06:20	2.5M
quickstart_install.log	2020-03-28 06:20	96K
collect_logs.sh	2020-03-28 05:04	5.0K
role-vars.yaml	2020-03-28 05:01	3
get_hash_log.log	2020-03-28 05:01	5.3K
hash_info.sh	2020-03-28 05:01	296
hostvars-variables.yaml	2020-03-28 05:00	62K
zuul-variables.yaml	2020-03-28 05:00	10K

README for Quickstart Logs

Links to important log files

undercloud/var/log/extra/errors.txt.txt.gz - the concatenation of all the errors on any node in a single file
undercloud/home/zuul/ - the source and log output of all templated shell scripts
undercloud/var/log/ - directories and files copied from /var/log on the undercloud. If other overcloud/subnodes exist, similar $node/var/log directories will also exist in these logs.
undercloud/var/log/extra/ - extra system details like package list, and cpu info gathered from the undercloud
undercloud/var/log/extra/rpm-list.txt.gz - rpms installed to the undercloud, container rpms can be found in extra//containers/$container/info.log
undercloud/var/lib/mistral - output of all ansible used by config-download to drive the overcloud deployment
undercloud/etc/yum.repos.d/ - link to the yum.repos.d directory
undercloud/etc/yum.repos.d/delorean.repo - link to the delorean.repo ( the build )

Links container log files

undercloud/var/log/containers/ - the system logs for each container
undercloud/var/log/extra/podman/ - the podman container setup configuration and setup logs
undercloud/var/log/extra/docker/ - the docker container setup configuration and setup logs
undercloud/var/log/tripleo-container-image-prepare.log - the container download, container update and provision log

Links to package builds and tempest

delorean_logs - the source code change built into rpm and the build logs are here
undercloud/var/log/tempest/ - Tempest run results logs
undercloud/home/zuul/tempest/etc/ - tempest.conf and tempest blacklist and whitelist tests
stackviz - stackviz tempest test results

Links to puppet logs

undercloud/var/lib/config-data/ - the configuration files for each openstack service
undercloud/etc/puppet/hieradata/ - the hieradata of the deployment, service-configs, net and vip info

undercloud/var/lib/container-puppet - puppet config for each container

Please refer to the recreation instructions

Note: These logs are only available in jobs that use OVB.

OpenStack Virtual Baremetal Documentation
baremetal boot log - all the overcloud nodes should have boot logs associated e.g. baremetal_1, baremetal_2 etc.
baremetal controller log, a.k.a BMC
logs collected from overcloud-controller-[0-2] node
logs collected from the overcloud-novacompute-[0-1] node

Check the base directory for a file called *failure_reason* for automatic failure detection. If no known error has been found the file will be named "No_failure_reason_found".

Most of the undercloud and overcloud deployment log files can be found in undercloud/home/zuul tracebacks and other errors are collected in the following log per node:

undercloud/var/log/extra/errors.txt.txt.gz - the concatenation of all the errors on any node in a single file

Next check the console log and search for PLAY RECAP. There are sometimes multiple ansible runs in a job, usually the last one is the relevant.
If no PLAY RECAP text is found that usually means an infra failure before Quickstart could even start.
If this is a different Ansible error, that could mean either an infra problem (often has UNREACHABLE in the line) or a bug in Quickstart.

Ask on #tripleo to get help or open a bug on Launchpad. Add the "ci" tag if it's a CI issue and "quickstart" if you suspect that the bug is in Quickstart itself.

Finally try rechecking or asking on #tripleo

The logs contain files showing variables used in the job run.

undercloud/var/log/extra/dump_variables_vars.json.txt.gz - contains the variables used in the running the actual test
releases.sh - the output of the script setting release-related variables
playbook_executions.log - prints out the complete commands, with all expanded arguments, to run the playbooks

Upstream OpenStack Health, Elastic Search and Kibana

http://status.openstack.org/elastic-recheck/ - A tool to track the impact of known bugs in OpenStack CI
http://logstash.openstack.org - filter in details in the log files from OpenStack CI
OpenStack Health - upstream job results by project

Tools that will help you spot a trend in TripleO CI

Job Exploration - check the job history across upstream clouds
zuul job filter - zuul's job filter per job
http://cistatus.tripleo.org - Overall check job status
http://cistatus.tripleo.org/gates/ - Overall gate job status

Tools to compare one job to another.

jcomparison - A tool to compare results from one job to another
log reduce - A tool that uses AI features to reduce the noise in logs and present only what is needed for debug

As a debugging step, a job can be run manually with '-dryrun' appended to the job name. When the "playbook dry run" option is invoked, the playbooks will not execute and collect logs will not run but certain log files, including 'toci_env_args_output.log', which contains the environment variables used in the job, and playbook_executions.log will still be produced in the logs directory for inspection. This option serves to assist with debugging and to test the testing scripts themselves.