Debugging CI builds in Travis

6 days ago | 3 mins | debugging, travis-ci, development
Featured image
Photo by Marten Newhall on Unsplash

Sometimes a Travis job fails and after looking through the output you have no idea why. The things are getting even more mysterious if the script works fine on the local machine. I’ve been there recently when had to debug a job running E2E tests.

There is a solution on how to debug a Travis job locally in docker container but there is a better and more reliable way - running Travis job in debug mode.

Launching Travis job in debug mode

To start a job in debug mode it should be enabled for GitHub repository. Things are way easier if your repository is private, debug mode is enabled there by default. Just navigate to the job page in the web UI and click the Debug job button at the top right corner to start.

For public repositories you have to send a request to support@travis-ci.com and specify the list repositories where you’d like to make debug mode available. It can take some time but needs to be done only once for the repository so it shouldn’t be a problem. I was lucky enough and my request was handled pretty fast (within several minutes).

Note: Switching repository from private to public will disable debug mode mentioned above. So you’ll have to send a request to support.

Once the debugging is enabled for the repository the only way to launch a job in a debug mode for a public repository is via API call. Here is the example with curl:

curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -H "Travis-API-Version: 3" \
  -H "Authorization: token ${TRAVIS_TOKEN}" \
  -d "{\"quiet\": true}" \
  https://api.travis-ci.com/job/${JOB_ID}/debug

As you can see there are two things required to send such a request:

  • Travis token. You can grab one from Profile > Settings > Settings tab on https://travis-ci.com.
  • Job id. It can be taken from the Travis job page url - the numeric part at the end of the url. E.g. 123456789 for the url https://travis-ci.com/github/.../jobs/123456789.

Little hack: You can add the utility function to your ~/.bashrc:

travis_debug() {
  if [ $# -eq 0 ]; then
    echo "Job id is required"
    return -1;
  fi
  JOB_ID=$1
  curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -H "Travis-API-Version: 3" \
  -H "Authorization: token ${TRAVIS_TOKEN-$2}" \
  -d "{\"quiet\": true}" \
  https://api.travis-ci.com/job/${JOB_ID}/debug
}

Now to launch the job in debug mode run in the terminal:

TRAVIS_TOKEN=<your token here> travis_debug <job id>

// or

travis_debug <job id> <your token here>

Debugging the job

Once the job started in debug mode go to the job log and take ssh connection string, similar to this one:

...
Use the following SSH command to access the interactive debugging environment:
ssh DwBhYvwgoBQ2dr7iQ5ZH34wGt@ny2.tmate.io
...

Paste it into the terminal as instructed to connect to the Travis container running the job. Starting from here you are connected to the fresh instance and can do almost anything you want to debug the script like run commands one by one, change environmental variables, edit scripts or run custom commands and scripts.

To make things a bit easier Travis provides handy functions of format travis_run_<job phase> e.g. travis_run_before_install, to run scripts of a specific phase as per job config. See the full list of functions here.

It’s worth mentioning that the debug session is tmate session rather than fully functional terminal. Which means you can’t use things like scp to download files to your machine so other workarounds are required in case you need it.

Another unpleasant thing is that closing the session terminates the job and you’ll have to start debugging from scratch in case the session is accidentally closed. Follow these instructions if you need to keep debug session output on exit.

Besides that debug mode is super handy and can save you a ton of time while debugging and fixing failed jobs.

Happy debugging!

#debugging#travis-ci#development