Ansible Lookup Plugin?

scottsb
scottsb
Community Member

Has anybody worked on creating a lookup plugin for Ansible using the new CLI?

I may work on one myself if not, but I don't want to duplicate work that's already done (and open sourced) either. It seems like the Lastpass lookup plugin that's already distributed with Ansible could be used as a starting point:
https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/lookup/lastpass.py


1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

«1

Comments

  • I'm not aware of such a plugin being developed by anyone. Ansible seems neat though, I hadn't heard of it before today.

    Rick

  • scottsb
    scottsb
    Community Member

    FYI, I've gone ahead and built this myself. Will be open sourcing it shortly and/or contributing it to Ansible core. Just a heads up to anybody who finds this thread.

  • That's awesome. We'd love to hear about it when it's available.

    Rick

  • scottsb
    scottsb
    Community Member

    I have submitted this as a pull request, hopefully to be distributed with Ansible:
    https://github.com/ansible/ansible/pull/37207

    If it gets rejected for some reason, then I will bundle it up as a plugin for others to install via Galaxy.

  • Woo! Here's to hoping they merge the pull request. :)

    Rick

  • cohix
    cohix
    1Password Alumni

    @scottsb I reviewed the PR, looks really great :)

  • sam.doran
    sam.doran
    Community Member
    edited March 2018

    I’m an Ansible Core developer. I'll review the PR and see about getting it merged.

  • sam.doran
    sam.doran
    Community Member
    edited March 2018

    Since you can also edit items with the 1Password CLI it would also be feasible to create modules to create, update, and delete 1Password records. This would make 1Password a first-class secret management tool alongside other "big name" secret stores. I like the implications of this.

  • Unfortunately we can't actually edit items yet with op. Most of the lower level work needed to make that happen is there (it's so similar to creation). One of the things we've been having trouble with is just how to expose this to users. To be honest, I'm not a big fan of how op create item works, and I think we need to reconsider how it works. Every item has effectively 3 parts: the item metadata (createdAt, updatedAt, etc...) which isn't encrypted, the item overview (contains things like title, urls, tags) which is encrypted, and item details (everything really interesting goes in there) which is also encrypted. Currently op create item has you give it only the details, and then specify a few of the fields you can put in the overview as flags.

    My gut says that we need to redo things all the way back to op get template and have that command return you a structure that looks just like op get item would, except empty. Then you'd pass that whole structure (which includes metadata, overview, details) over to op create item. If we did that, then op update item could work exactly the same and the world would be a beautiful place.

    Rick

  • MrC
    MrC
    Volunteer Moderator

    Yes! @rickfillion ,

    I've been contemplating the road path to providing my converter suite with the ability to submit entries directly via op create item, but I don't think the CLI is ready for that yet. There does need to be a structure that supports packaging the metadata, key field data, notes, and perhaps even attachments (+ internal linking).

  • God that'd be hot, @MrC.

    Rick

  • scottsb
    scottsb
    Community Member
    edited May 2018

    FYI, the Ansible lookup plugin for 1PW was merged today. Thanks to @sam.doran for help with that. Planned for Ansible 2.6, which is anticipated to release as stable on June 28.

  • Way to go @scottsb and @sam.doran! That's awesome.

    Rick

  • eby
    eby
    Community Member

    Just want to thank @scottsb and @sam.doran for this. Something like hashicorp vault is overkill for some of the things I'm doing and this makes my setup a lot easier.

  • sam.doran
    sam.doran
    Community Member

    @eby Glad to hear!

    I added login capability to the lookups for Ansible 2.7 (release date is 2018-10-04), plus a community member contributed a facts module to Ansible, allowing 1Passwod items to be used as Ansible facts.

    Right now I’m working on unifying the interface across them all so it feels consistent.

  • cohix
    cohix
    1Password Alumni

    That's super cool!

  • mickael
    mickael
    Community Member

    Hey @scottsb, @sam.doran,

    I am looking for some feedback on how 1Password works with ansible.

    We are currently in the way of leaving LastPass due to issues we encounter with Ansible and our other DevOps scripts. Mainly we are facing two huge issues with LastPass:

    • ID of items change when you move them from one vault to another
    • SSL verification fails randomly causing various CLI calls to fail and so causing the script where the lastpass cli call is done to fail also. This could be the run of our Ansible playbooks or also the lost of some passwords due to synchronization failures.

    The "good" thing with LastPass is that his CLI have good performance, around 130ms to query our vault with 1500 entries.

    So, how does 1Password plays with your playbook. We manage the hosting infrastructure of our customers with Ansible and have hundred of lookup query in our playbook. So we are looking for a fast and reliable solution.

    Do you advice 1Password to handle such an ansible usage ?

    Do you benchmark the time of a lookup query ?

    Are you using 1Password with the cloud service or with a local vault ?

    Thanks for your feedbacks,

  • sam.doran
    sam.doran
    Community Member

    Do you advice 1Password to handle such an ansible usage?

    1Password would probably work fine but you would need to change your strategy. If you have hundreds of lookups per play, I would recommend using the onepassword_facts module to get everything you need in one query, rather than making hundreds of queries. Note that once those items are stored as facts, you will need to safeguard that data. Ansible does not treat data from 1Password any different than other variables. Be especially careful if you have fact caching enabled. (This is not unique to 1Password — this applies to data from any lookup plugin).

    Do you benchmark the time of a lookup query?

    I did during initial development but don't recall the exact numbers. Rather than focus on total time per query I would find ways to do fewer queries. I just did a few tests on macOS 10.14.4 with op version 0.5.5 by running time op get item foo and the results range between 0.78 and 0.95

    Are you using 1Password with the cloud service or with a local vault?

    I did my testing and development using the cloud service. I never tested using a local vault.

    If you need a truly high performance secret store, you could also look at Hashicorp Vault. Ansible has a lookup plugin for that as well.

  • scottsb
    scottsb
    Community Member

    Are you using 1Password with the cloud service or with a local vault ?

    Sam's answers are good, but one key note: the Ansible plugin uses the op command line tool under the surface, and that only works with the cloud service, not local vaults.

  • mickael
    mickael
    Community Member

    @sam.doran Thanks a lot for that quick and detailed answer. You exposed a really interesting strategy, I wasn't aware of onepassword_facts this could be definitely the solution. I will dig into this. Thanks so much ;)

    However, I am really surprised by the performance of your CLI, which version are you using ? I have exposed my bencharks on this post https://discussions.agilebits.com/discussion/comment/502630/ and I am around 4-5s :dizzy: Are you using a future unreleased version of the CLI ?

    Regarding Hashicorp Vault, yeah, this could be the solution for ansible, but this won't fill all the others features that can give 1Password: outstanding UI, apps everywhere, usable by non technical users, pricing as we don't want to handle the hosting ourselves...

    @scottsb thanks for your clarification, we don't intend to use local vaults especially if they can't synchronize with online service

  • sam.doran
    sam.doran
    Community Member

    Not sure why you're seeing such long times. Here is my result from running your test. Version 0.5.5 does seem to be a bit faster than previous versions.


  • mickael
    mickael
    Community Member

    @sam.doran Correct me if I am wrong, I have inspected the onepassword_facts source code. But as far as I understand, each fact is still gathered from a separate call of the op cli in the get_raw method.

    So, the profit is "only" to prevent calls to the lookup module each time the group/host config need to be accessed. This is fine but with a response of 4s by each op query the run of ansible would be delayed by 20 minutes for 300 entries. Still unacceptable sadly :(

  • sam.doran
    sam.doran
    Community Member

    @mickael It looks like that is the case, so that would only help if you ran the fact task and cached the results (which entails some risk).

  • mickael
    mickael
    Community Member

    @sam.doran I am currently thinking about a way to use the native ansible vault tocache 1Password secrets.

    Ideally, if a var is not defined, ansible would call a custom plugin and try to lookup the value in 1Password depending on the naming scheme for example. If the value can be grabbed, it will be stored in the vault.

    Do you have any advices if this could be possible ? I identified some Python projects that helps in updating Ansible vault but currently have no clue on how to handle the fallback if the var is not defined.

    A workaround could be to use a specific lookup plugin that will search in the vault before querying 1Password, but at the expense of a more verbose syntax.

  • jamesdh
    jamesdh
    Community Member

    It looks like this is currently limited to items and not capable of retrieving documents at the moment. If I were to submit a PR adding that functionality, would the preference be to amend the current onepassword lookup, or to create a separate onepassword_doc lookup specifically for documents?

    The semantics are similar but not matching...

    usage: op get document <document> [--vault=<vault>] > <filename>
    vs...
    usage: op get item <item> [--vault=<vault>] [--include-trash]

  • jamesdh
    jamesdh
    Community Member

    Forgot to CC @scottsb @cohix above...

  • scottsb
    scottsb
    Community Member
    edited April 2019

    @sam.doran is the Ansible core maintainer amongst us, so he would be best suited to answer. :-)

  • cohix
    cohix
    1Password Alumni

    That sounds like it would be entirely doable :)

  • sam.doran
    sam.doran
    Community Member

    @jamesdh There is an open pull request to add a onepassword_doc lookup plugin. I had some issues with it in my testing and left some feedback in the review. I'm waiting on the PR author to address those comments.

  • mickael
    mickael
    Community Member
    edited April 2019

    FYI,

    I just published the Ansible lookup module that uses my 1Password local database search module.

    I don't have finished to migrate to 1Password so I didn't use it in production so far but Molecule tests are passing ;)

    Code is here: https://github.com/mickaelperrin/ansible-onepassword-local-lookup-plugin

    ping @sam.doran

This discussion has been closed.