Prevent simultaneous deploys with Ansible

ansible wait_for
ansible sleep
ansible handlers
ansible until
ansible debug handlers
ansible file watcher
timeout in ansible playbook
ansible task timeout

Anyone on my team can SSH into our special deploy server, and from there run an Ansible playbook to push new code to machines.

We're worried about what will happen if two people try to do deploys simultaneously. We'd like to make it so that the playbook will fail if anyone else is currently running it.

Any suggestions for how to do this? The standard solution is to use a pid file, but Ansible does not have built-in support for these.

You could write a wrapper for ansible commands like so :

ansible-playbook() {
  lock="/tmp/ansible-playbook.lock"

  # Check if lock exists, return if yes
  if [ -e $lock ]; then
    echo "Sorry, someone is running already ansible from `cat $lock`"
    return
  fi

  # Install signal handlers
  trap "rm -f $lockfile; trap - INT TERM EXIT; return" INT TERM EXIT

  # Create lock file, saving originating IP
  echo $SSH_CLIENT | cut -f1 -d' ' > $lock

  # Run ansible with arguments passed at the command line
  `which ansible-playbook` "$@"

  # Remove lock file
  rm $lock

  # Remove signal handlers
  trap - INT TERM EXIT
}

Define this function in the ~/.bashrc of your users on the deploy box and you're set. You can do the same for the ansible command to if you wish, but given the question I'm not sure it's required.

EDIT: Rewrote with signal handler to prevent lock file dangling around if users hit Ctrl-C.

EDIT2: Fixed typo

How do I prevent two ansible jobs to run against the same machine , While a lock file is a technical solution, I'd like to instead propose a cultural one: a vocalized "deploy lock". Many development teams have an  Which is to say that, by default, Tower in OpenShift (when configured entirely for cpu-based algorithm) can run at most 6 simultaneous forks. The other value that can be tuned: task_mem_request = 2 - This is the amount of memory to dedicate ( in gigabytes ) .

Prevent simultaneous deploys with Ansible - ansible, Anyone on my team can SSH into our special deploy server, and from there run an Ansible playbook to push new code to machines. We're worried about what  Role documentation¶ afs-release¶. afs-release. Install the script and related bits and pieces for periodic release of various AFS volumes. This role is really only intended to be run on the mirror-update host, as it uses the ssh-key installed by that host to run vos release under -localauth on the remote AFS servers.

I put this in my main playbook, after

    hosts: all. 

lock_file_path: This is a file whose existence indicates that there is a currently running ansible deploy, or there was a deploy before, that got aborted for some reason.

force_ignore_lock: This defaults to false, reset by an option flag that you can set in a command-line wrapper to ansible. It enables ansible to continue with the deploy.

What this does
pre_tasks

The first pre_task checks if the lock_file_path exists, and records the result in a register called lock_file.

The next task then checks to see if the file exists, and if the person deploying has chosen to ignore it (hopefully after communicating with other team-mates). If not, the job fails, with a helpful error message.

If the user chooses to move on with the deploy even if there was a lock_file, the next task deletes the previously created lock_file and creates a new one. The other tasks in the selected role then continue happily.

post_tasks

This is a hook called immediately after all the tasks in the deploy have been completed. The task here deletes the lock_file, enabling the next person to deploy happily, without any issues.

vars:
  lock_file_path=/tmp/ansible-playbook-{{ansible_ssh_user}}.lock

pre_tasks: 
   - stat: path={{lock_file_path}}
     register: lock_file

   - fail: msg="Sorry, I found a lockfile, so I'm assuming that someone was already running ansible when you started this deploy job. Add -f to your deploy command to forcefully continue deploying, if the previous deploy was aborted."
   when: lock_file.stat.exists|bool and not force_ignore_lock|bool

   - file: path={{lock_file_path}} state=absent
     sudo: yes
     when: "{{force_ignore_lock}}"

   - file: path={{lock_file_path}} state=touch
     sudo: yes

post_tasks:
   - file: path={{lock_file_path}} state=absent
     sudo: yes

16. Jobs, Tower limits the number of simultaneous jobs that can run based on the amount of They block all other jobs and must be run on their own to avoid conflict. Complete – Ansible does three things in one and does them very well. Ansible’s ‘batteries included’ approach means you have everything you need in one complete package. Efficient – No extra software on your servers means more resources for your applications. Also, since Ansible modules work via JSON, Ansible is extensible with modules written in a programming language you already know.

Have you considered setting maxsyslogins in limits.conf? You can restrict this by group.

# for a group called 'deployers'
@deployers        -       maxsyslogins      1

This is quite a bit more severe than what you asked for. You might want to try it on a VM first. Note that no-one from deployers will have access if there are any other users on the system at all, the 1 limit doesn't just count deployers. Also, if as a user you multiplex your ssh connections (ControlMaster auto), you will still be able to log in multiple times; it's other users who'd be locked out.

Delegation, Rolling Updates, and Local Actions, This in particular is very applicable when setting up continuous deployment traffic on load balancers (must be turned off simultaneously); gracefully stop the  Preamble to avoid potential comments on my choice of using alternatives instead of forking out truckloads of dosh to Ansible for an opensource application. I happily contribute to the Ansible project via GitHub when I find things that need fixing. So personally I feel I'm supporting Ansible in general without needing to buy Ansible Tower.

You can use the flock command, which will wrap your command with a filesystem-based flock(2):

$ flock /tmp/ansible-playbook.lock ansible-playbook foo bar baz

Wrap that however it suits your users best. This will leave a persistent lockfile in /tmp, but note that it is not safe to delete it[1]. It's atomic, and very simple.

[1]
A: flock /tmp/foo.lock -c "echo running; sleep 5; rm /tmp/foo.lock"
B: flock /tmp/foo.lock -c "echo running; sleep 5; rm /tmp/foo.lock"
   B blocks waiting for lock on /tmp/foo.lock
A: Finish, deleting /tmp/foo.lock
B: Runs, using lock on now deleted /tmp/foo.lock
C: flock /tmp/foo.lock -c "echo running; sleep 5; rm /tmp/foo.lock"
   Creates new /tmp/foo.lock, locks it and runs immediately, parallel with B

Ansible Performance Tuning (for Fun and Profit), Further, the push architecture can actually help avoid some of the run dozens of times a day if you are practicing Continous Deployment. Appendix D: Security¶. Security is one of the top priorities within OpenStack-Ansible (OSA), and many security enhancements for OpenStack clouds are available in deployments by default. This appendix provides a detailed overview of the most important security enhancements.

7 Tips To Turbo-charge Your Ansible, Ansible is a very popular orchestration, deployment, and that the number of servers where the playbook executes simultaneously is 5. The intended use here is to run this role in Ansible Tower. Ansible Tower notifications can be configured for job failure via email, Slack or other methods. This role runs in Ansible, Ansible Engine or Ansible Tower. We’ve condensed the script and created a fully idempotent role that can enforce the desired state of the sudoers file.

Lessons from using Ansible exclusively for 2 years., This post doesn't compare Ansible to Puppet / Chef. One of the first projects I used Ansible for was to simultaneously deploy and remove a  Red Hat Ansible. Ansible is an open source community project sponsored by Red Hat, it's the simplest way to automate IT. Ansible is the only automation language that can be used across entire IT teams from systems and network administrators to developers and managers. About Us Our Story Press Center Careers

Parallel playbook execution · Issue #31 · ansible/proposals · GitHub, Proposal: Enhance Ansible to allow playbooks to be run in parallel. would be executed simultaneously and end-deployment-notifications would only Those two use-cases above were preventing me to allow for proper  Also, running a playbook against a single device is not a huge efficiency gain over making the same change manually. The next step to harnessing the full power of Ansible is to use an inventory file to organize your managed nodes into groups with information like the ansible_network_os and the SSH user. A fully-featured inventory file can serve as the source of truth for your network.

Comments
  • Wow. This script is going to mostly prevent two instances running, but not fully, due to the time gap between testing and creating the lock file. You need an atomic operation to have real protection, and though I am out of date with *nix, I think a script cannot do this. I presume RunDeck mentioned in next answer may have what's required, using IPC (semaphores, etc.).
  • In theory, yes, there is no 100%. You could lower the gap more doing something like [ -e $lock ] || (echo $SSH_CLIENT | cut -f1 -d' ' > $lock) (and adjust the script accordingly) which takes less than a millisecond. In practice, sounds quite reasonable. Or write the lock file with restricted perms so another user can not overwrite it. All in all of course, this is a hack at best. Rundeck looks cool.
  • flock with -c is an option here, like so flock -x $HOME/.ansible_lock -c 'ansible-playbook ...'
  • Are there any details online about how you have this set up?
  • @thisjustin Not fully as yet - I've been sidetracked by work lately, so my blog isn't quite at that point as yet - rundeck has the option to configure a job to only run if it isn't already running in its configuration per job and returns the appropriate 4XX code to say if a job is already running if that option is set (if you are using the API thats an important piece of info to know)
  • RunDeck looks like jenkins to me.
  • If the ansible execution is interrupted you may end-up with a permanently locked machine. The existence of a lock file is not enough, it needs a way to detect if that's a stale lock or not.
  • This is why I added is: when: lock_file.stat.exists|bool and not force_ignore_lock|bool. This uses the force_ignore_lock variable that can be set in a command-line flag to a wrapper script written in e.g. python using click. Example: ``
  • Example: @deploy.command() @click.option('--force', help="Ignore the lock file that prevents parallel deploys, \ if a previous deploy was aborted.", is_flag=True, default=False) def backend(api_server, force): """Deploys the backend application.""" if force: ignore_lock = 'true' else: ignore_lock = 'false' extra_vars = { 'force_ignore_lock': ignore_lock } call_ansible(api_server, 'api_server.yml', json.dumps(extra_vars))
  • I had actually Googled for something to do exactly that as a possible solution, but couldn't find it. Thanks!