Ansible Lookup Plugin?

Has anybody worked on creating a lookup plugin for Ansible using the new CLI?

I may work on one myself if not, but I don't want to duplicate work that's already done (and open sourced) either. It seems like the Lastpass lookup plugin that's already distributed with Ansible could be used as a starting point:
https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/lookup/lastpass.py


1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

«1

Comments

  • rickfillionrickfillion Junior Member

    Team Member

    I'm not aware of such a plugin being developed by anyone. Ansible seems neat though, I hadn't heard of it before today.

    Rick

  • FYI, I've gone ahead and built this myself. Will be open sourcing it shortly and/or contributing it to Ansible core. Just a heads up to anybody who finds this thread.

  • rickfillionrickfillion Junior Member

    Team Member

    That's awesome. We'd love to hear about it when it's available.

    Rick

  • I have submitted this as a pull request, hopefully to be distributed with Ansible:
    https://github.com/ansible/ansible/pull/37207

    If it gets rejected for some reason, then I will bundle it up as a plugin for others to install via Galaxy.

  • rickfillionrickfillion Junior Member

    Team Member

    Woo! Here's to hoping they merge the pull request. :)

    Rick

  • cohixcohix

    Team Member

    @scottsb I reviewed the PR, looks really great :)

  • sam.doransam.doran Junior Member
    edited March 2018

    I’m an Ansible Core developer. I'll review the PR and see about getting it merged.

  • sam.doransam.doran Junior Member
    edited March 2018

    Since you can also edit items with the 1Password CLI it would also be feasible to create modules to create, update, and delete 1Password records. This would make 1Password a first-class secret management tool alongside other "big name" secret stores. I like the implications of this.

  • rickfillionrickfillion Junior Member

    Team Member

    Unfortunately we can't actually edit items yet with op. Most of the lower level work needed to make that happen is there (it's so similar to creation). One of the things we've been having trouble with is just how to expose this to users. To be honest, I'm not a big fan of how op create item works, and I think we need to reconsider how it works. Every item has effectively 3 parts: the item metadata (createdAt, updatedAt, etc...) which isn't encrypted, the item overview (contains things like title, urls, tags) which is encrypted, and item details (everything really interesting goes in there) which is also encrypted. Currently op create item has you give it only the details, and then specify a few of the fields you can put in the overview as flags.

    My gut says that we need to redo things all the way back to op get template and have that command return you a structure that looks just like op get item would, except empty. Then you'd pass that whole structure (which includes metadata, overview, details) over to op create item. If we did that, then op update item could work exactly the same and the world would be a beautiful place.

    Rick

  • MrCMrC Community Moderator

    Yes! @rickfillion ,

    I've been contemplating the road path to providing my converter suite with the ability to submit entries directly via op create item, but I don't think the CLI is ready for that yet. There does need to be a structure that supports packaging the metadata, key field data, notes, and perhaps even attachments (+ internal linking).

  • rickfillionrickfillion Junior Member

    Team Member

    God that'd be hot, @MrC.

    Rick

  • scottsbscottsb
    edited May 2018

    FYI, the Ansible lookup plugin for 1PW was merged today. Thanks to @sam.doran for help with that. Planned for Ansible 2.6, which is anticipated to release as stable on June 28.

  • rickfillionrickfillion Junior Member

    Team Member

    Way to go @scottsb and @sam.doran! That's awesome.

    Rick

  • Just want to thank @scottsb and @sam.doran for this. Something like hashicorp vault is overkill for some of the things I'm doing and this makes my setup a lot easier.

  • sam.doransam.doran Junior Member

    @eby Glad to hear!

    I added login capability to the lookups for Ansible 2.7 (release date is 2018-10-04), plus a community member contributed a facts module to Ansible, allowing 1Passwod items to be used as Ansible facts.

    Right now I’m working on unifying the interface across them all so it feels consistent.

  • cohixcohix

    Team Member

    That's super cool!

  • Hey @scottsb, @sam.doran,

    I am looking for some feedback on how 1Password works with ansible.

    We are currently in the way of leaving LastPass due to issues we encounter with Ansible and our other DevOps scripts. Mainly we are facing two huge issues with LastPass:

    • ID of items change when you move them from one vault to another
    • SSL verification fails randomly causing various CLI calls to fail and so causing the script where the lastpass cli call is done to fail also. This could be the run of our Ansible playbooks or also the lost of some passwords due to synchronization failures.

    The "good" thing with LastPass is that his CLI have good performance, around 130ms to query our vault with 1500 entries.

    So, how does 1Password plays with your playbook. We manage the hosting infrastructure of our customers with Ansible and have hundred of lookup query in our playbook. So we are looking for a fast and reliable solution.

    Do you advice 1Password to handle such an ansible usage ?

    Do you benchmark the time of a lookup query ?

    Are you using 1Password with the cloud service or with a local vault ?

    Thanks for your feedbacks,

  • sam.doransam.doran Junior Member

    Do you advice 1Password to handle such an ansible usage?

    1Password would probably work fine but you would need to change your strategy. If you have hundreds of lookups per play, I would recommend using the onepassword_facts module to get everything you need in one query, rather than making hundreds of queries. Note that once those items are stored as facts, you will need to safeguard that data. Ansible does not treat data from 1Password any different than other variables. Be especially careful if you have fact caching enabled. (This is not unique to 1Password — this applies to data from any lookup plugin).

    Do you benchmark the time of a lookup query?

    I did during initial development but don't recall the exact numbers. Rather than focus on total time per query I would find ways to do fewer queries. I just did a few tests on macOS 10.14.4 with op version 0.5.5 by running time op get item foo and the results range between 0.78 and 0.95

    Are you using 1Password with the cloud service or with a local vault?

    I did my testing and development using the cloud service. I never tested using a local vault.

    If you need a truly high performance secret store, you could also look at Hashicorp Vault. Ansible has a lookup plugin for that as well.

  • Are you using 1Password with the cloud service or with a local vault ?

    Sam's answers are good, but one key note: the Ansible plugin uses the op command line tool under the surface, and that only works with the cloud service, not local vaults.

  • @sam.doran Thanks a lot for that quick and detailed answer. You exposed a really interesting strategy, I wasn't aware of onepassword_facts this could be definitely the solution. I will dig into this. Thanks so much ;)

    However, I am really surprised by the performance of your CLI, which version are you using ? I have exposed my bencharks on this post https://discussions.agilebits.com/discussion/comment/502630/ and I am around 4-5s :dizzy: Are you using a future unreleased version of the CLI ?

    Regarding Hashicorp Vault, yeah, this could be the solution for ansible, but this won't fill all the others features that can give 1Password: outstanding UI, apps everywhere, usable by non technical users, pricing as we don't want to handle the hosting ourselves...

    @scottsb thanks for your clarification, we don't intend to use local vaults especially if they can't synchronize with online service

  • sam.doransam.doran Junior Member

    Not sure why you're seeing such long times. Here is my result from running your test. Version 0.5.5 does seem to be a bit faster than previous versions.

    > sysctl -n machdep.cpu.brand_string
    Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    
    > op --version
    0.5.5
    
    > for i in (seq 1 20)
          time op get item 'Tower Demo' > /dev/null
      end
            0.89 real         0.25 user         0.10 sys
            0.78 real         0.25 user         0.09 sys
            1.19 real         0.24 user         0.09 sys
            0.81 real         0.24 user         0.09 sys
            0.97 real         0.25 user         0.09 sys
            1.06 real         0.25 user         0.10 sys
            0.87 real         0.26 user         0.09 sys
            0.83 real         0.23 user         0.09 sys
            1.12 real         0.24 user         0.09 sys
            0.84 real         0.24 user         0.09 sys
            0.99 real         0.25 user         0.10 sys
            0.83 real         0.25 user         0.09 sys
            0.93 real         0.24 user         0.08 sys
            0.92 real         0.24 user         0.10 sys
            0.84 real         0.25 user         0.09 sys
            0.81 real         0.25 user         0.09 sys
            1.08 real         0.25 user         0.09 sys
            1.01 real         0.27 user         0.11 sys
            0.85 real         0.24 user         0.09 sys
            0.98 real         0.24 user         0.09 sys
    
  • @sam.doran Correct me if I am wrong, I have inspected the onepassword_facts source code. But as far as I understand, each fact is still gathered from a separate call of the op cli in the get_raw method.

    So, the profit is "only" to prevent calls to the lookup module each time the group/host config need to be accessed. This is fine but with a response of 4s by each op query the run of ansible would be delayed by 20 minutes for 300 entries. Still unacceptable sadly :(

  • sam.doransam.doran Junior Member

    @mickael It looks like that is the case, so that would only help if you ran the fact task and cached the results (which entails some risk).

  • @sam.doran I am currently thinking about a way to use the native ansible vault tocache 1Password secrets.

    Ideally, if a var is not defined, ansible would call a custom plugin and try to lookup the value in 1Password depending on the naming scheme for example. If the value can be grabbed, it will be stored in the vault.

    Do you have any advices if this could be possible ? I identified some Python projects that helps in updating Ansible vault but currently have no clue on how to handle the fallback if the var is not defined.

    A workaround could be to use a specific lookup plugin that will search in the vault before querying 1Password, but at the expense of a more verbose syntax.

  • It looks like this is currently limited to items and not capable of retrieving documents at the moment. If I were to submit a PR adding that functionality, would the preference be to amend the current onepassword lookup, or to create a separate onepassword_doc lookup specifically for documents?

    The semantics are similar but not matching...

    usage: op get document <document> [--vault=<vault>] > <filename>
    vs...
    usage: op get item <item> [--vault=<vault>] [--include-trash]

  • Forgot to CC @scottsb @cohix above...

  • scottsbscottsb
    edited April 24

    @sam.doran is the Ansible core maintainer amongst us, so he would be best suited to answer. :-)

  • cohixcohix

    Team Member

    That sounds like it would be entirely doable :)

  • sam.doransam.doran Junior Member

    @jamesdh There is an open pull request to add a onepassword_doc lookup plugin. I had some issues with it in my testing and left some feedback in the review. I'm waiting on the PR author to address those comments.

  • mickaelmickael
    edited April 26

    FYI,

    I just published the Ansible lookup module that uses my 1Password local database search module.

    I don't have finished to migrate to 1Password so I didn't use it in production so far but Molecule tests are passing ;)

    Code is here: https://github.com/mickaelperrin/ansible-onepassword-local-lookup-plugin

    ping @sam.doran

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file