CLI secure-note utilities written in python

HippoMan
HippoMan
Community Member
edited May 2018 in CLI

After finally figuring out how to get the CLI working via shell-script wrappers, I have now written some python scripts for managing Secure Notes via the CLI ... currently, these work in version 0.4.1.

I chose python because it offers more programming capabilities than shell languages, such as a json module that eliminates the need for hacking around with jq. Also, I didn't choose go, because I have years of experience with python, but am still learning go.

These scripts can create, view, update, and delete Secure Notes. The only caveat is that by "delete", I mean "put them in the Trash". The Trash will still have to be emptied manually via the 1Password UI. Once the CLI offers an empty-the-Trash capability (please!!!), I can update these python programs so they don't leave lots of dangling garbage sitting around in the Trash.

As soon as I finish writing this message, I will post the scripts here, one by one, in this thread. Here's a summary ...

  1. op-item-exists -- python program: tests for the existence of any item in any vault (used internally in the other programs)
  2. op-get-note -- python program: retrieves the contents of any Secure Note
  3. op-put-note -- python program: creates or updates a Secure Note
  4. op-rm-note -- python program: deletes a Secure Note
  5. op-create-note -- shell script: very thin wrapper around op-put-note which only does creation of Secure Notes
  6. op-update-note -- shell script: very thin wrapper around op-put-note which only does updating of Secure Notes

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

Comments

  • HippoMan
    HippoMan
    Community Member

    op-item-exists -- check for item existence

    usage: op-item-exists [ -qQvV | --verbose | --quiet ] item [ optional 'op create item' arguments ]
    
      tests for the existence of an item, and prints out its uuid if found
    
      -v, -V, or --verbose mean to print out some error messages
      -q, -Q, or --quiet   mean to not print out the item's uuid if found
      note that --verbose and --quiet are independent; both can exist
    
      returns  0 if the item exists,
               N if it finds N (> 1) matched items,
              -1 if the item doesn't exist,
               1 in all other cases
    

    Python code ...

    #!/usr/bin/python3
    
    import sys
    sys.dont_write_bytecode = True
    
    import os
    import re
    import json
    import getopt
    
    sys.path.insert(0, '/usr/local/etc/python')
    from u import invoke
    
    prog    = os.path.basename(sys.argv[0])
    op      = '/usr/local/bin/op'
    multpat = re.compile(r'\(ERROR\)\s+multiple\s+items\b', re.M | re.DOTALL)
    nonepat = re.compile(r'\(ERROR\)\s+\bitem\s+.+?\s+not\s+found\b', re.M | re.DOTALL)
    
    def main():
        try:
            opts, args = getopt.getopt(sys.argv[1:], 'vVqQ', ['verbose', 'quiet'])
        except getopt.GetoptError as e:
            print('{}'.format(e))
            return 1
    
        quiet   = False
        verbose = False
        for o, a in opts:
            if o in ('-v', '-V', '--verbose'):
                verbose = True
            elif o in ('-q', '-Q', '--quiet'):
                quiet   = True
            else:
                return usage('invalid option: {}'.format(o))
    
    if len(args) < 1:
        return usage()
    
    item = args[0]
    
    rc, o, e = invoke([ op, 'get', 'item', item ] + args[1:])
    if rc == 0:
        resp = json.loads(o)
        if not quiet:
            print(resp['uuid'])
        return 0
    elif multpat.search(e):
        ndups = len(o.split('\n')) - 1
        if verbose:
            print('{} matching items: {}'.format(ndups, item))
        return ndups
    elif nonepat.search(e):
        if verbose:
            print('no matching items: {}'.format(item))
        return -1
    else:
        return rc
    
    def usage(msg=None):
        if msg:
            print('\n{}'.format(msg))
        print('''
    usage: {} [ -qQvV | --verbose | --quiet ] item [ optional 'op create item' arguments ]
    
      tests for the existence of an item, and prints out its uuid if found
    
      -v, -V, or --verbose mean to print out some error messages
      -q, -Q, or --quiet   mean to not print out the item's uuid if found
      note that --verbose and --quiet are independent; both can exist
    
      returns  0 if the item exists,
               N if it finds N (> 1) matched items,
              -1 if the item doesn't exist,
               1 in all other cases
    '''.format(prog))
        return -2
    
    if __name__ == '__main__':
        sys.exit(main())
    
  • HippoMan
    HippoMan
    Community Member

    op-get-note - retrieve Secure Note

    usage: op-get-note [ -qQ | --quiet ] item [ optional 'op get item' arguments ]
    
      -q, -Q, or --quiet mean to print no error messages
    
      if the item exists, returns 0 and prints the item to stdout,
      otherwise returns non-zero
    
      optional arguments could contain '--vault=XXX', for example
    

    Python code ...

    #!/usr/bin/python3
    
    import sys
    sys.dont_write_bytecode = True
    
    import os
    import json
    import getopt
    
    sys.path.insert(0, '/usr/local/etc/python')
    from u import invoke
    
    prog       = os.path.basename(sys.argv[0])
    op         = '/usr/local/bin/op'
    itemexists = '/usr/local/bin/op-item-exists'
    
    def main():
    
        try:
            opts, args = getopt.getopt(sys.argv[1:], 'qQ', ['quiet'])
        except getopt.GetoptError as e:
            print('{}'.format(e))
            return 1
    
        verbose = True
        for o, a in opts:
            if o in ('-q', '-Q', '--quiet'):
                verbose = False
            else:
                return usage('invalid option: {}'.format(o))
    
        if len(args) < 1:
            return usage()
    
        name = args[0]
        base = os.path.basename(name)
        rc, o, e = invoke([ itemexists, base ] + args[1:])
        if rc == 255:
            if verbose:
                print('not found: {}'.format(name))
            return -1
        elif rc > 1:
            if verbose:
                print('{} items found: {}'.format(rc, name))
        elif rc:
            return -1
    
        rc, o, e = invoke([ op, 'get', 'item', o.rstrip() ] + args[1:])
        if rc:
            if verbose:
                print(e)
            return rc
    
        jsonerror = 'invalid json: {}'.format(name)
        try:
            data = json.loads(o)
        except:
            if verbose:
                print(jsonerror)
            return 1
    
        details = data.get('details', None)
        if not details:
            if verbose:
                print(jsonerror)
            return 1
    
        notes = details.get('notesPlain', None)
        if notes is None:
            if verbose:
                print('invalid json: {}'.format(name))
            return 1
    
        sys.stdout.write('{}\n'.format(notes))
        sys.stdout.flush()
    
        return 0
    
    def usage(msg=None):
        if msg:
            print('\n{}'.format(msg))
        print('''
    usage: {} [ -qQ | --quiet ] item [ optional 'op get item' arguments ]
    
      -q, -Q, or --quiet mean to print no error messages
    
      if the item exists, returns 0 and prints the item to stdout,
      otherwise returns non-zero
    
      optional arguments could contain '--vault=XXX', for example
    '''.format(prog))
        return -2
    
    if __name__ == '__main__':
        sys.exit(main())
    
  • HippoMan
    HippoMan
    Community Member
    edited May 2018

    op-put-note - create or update a Secure Note

    usage: op-put-note [ -iIcCuU | --stdin | --create | --update ] item-name [ args ... ]
    
      'args' are optional 'op create item "Secure Note"' arguments
      which could contain '--vault=XXX' or '--tag=FOOBAR';
      but don't use '--title=XXX'
    
      returns 0 if the item has been created or updated, otherwise non-zero
    
      option -i, -I, or --stdin  means take item contents from stdin;
      option -c, -C, or --create means only create when item doesn't exist;
      option -u, -U, or --update means only update when item does exist;
      with neither --create nor --update, decide which to do based upon existence;
      --create and --update cannot both appear
    

    Python code ...

    #!/usr/bin/python3
    
    import sys
    sys.dont_write_bytecode = True
    
    import os
    import re
    import json
    import getopt
    
    sys.path.insert(0, '/usr/local/etc/python')
    from u import invoke, utf8decode
    
    prog       = os.path.basename(sys.argv[0])
    op         = '/usr/local/bin/op'
    itemexists = '/usr/local/bin/op-item-exists'
    titlepat   = re.compile(r'^--title=', re.I)
    
    def main():
    
        try:
            opts, args = getopt.getopt(sys.argv[1:], 'cCiIuU', ['create', 'stdin', 'update'])
        except getopt.GetoptError as e:
            print('{}'.format(e))
            return 1
    
        fromstdin   = False
        forcecreate = False
        forceupdate = False
        for o, a in opts:
            if o in ('-i', '-I', '--stdin'):
                fromstdin = True
            elif o in ('-c', '-C', '--create'):
                forcecreate = True
            elif o in ('-u', '-U', '--update'):
                forceupdate = True
            else:
                return usage('invalid option: {}'.format(o))
    
        if (forcecreate and forceupdate) or len(args) < 1:
            return usage()
    
        for a in args:
            if titlepat.search(a):
                return usage('illegal argument: {}'.format(a))
    
        item   = args[0]
        params = args[1:]
        base   = os.path.basename(item)
        cmd    = None
        data   = None
        exists = None
        uuid   = None
    
        rc, o, e = invoke([ itemexists, base ] + params)
        if rc == 0:
            if forcecreate:
                return 1
            exists = True
            uuid   = o.rstrip()
        elif rc == 255:
            if forceupdate:
                return 1
            exists = False
        else:
            print(e)
            return rc
    
        try:
            if fromstdin:
                data = utf8decode(sys.stdin.read())
            else:
                with open(item, 'r') as f:
                    data = utf8decode(f.read())
        except:
            return -1
    
        note = {
            'notesPlain': data,
            'sections':   []
        }
    
        jnote = json.dumps(note).rstrip()
    
        rc, o, e = invoke([ op, 'encode' ], input=jnote)
        if rc:
            return rc
    
        encoded = utf8decode(o.rstrip())
    
        rc, o, e = invoke([ op, 'create', 'item', 'Secure Note', encoded, '--title={}'.format(base) ] + params)
        if rc:
            return rc
    
        if exists:
            rc, o, e = invoke([ op, 'delete', 'item', uuid ] + params)
    
        return rc
    
    def usage(msg=None):
        if msg:
            print('\n{}'.format(msg))
        print('''
    usage: {} [ -iIcCuU | --stdin | --create | --update ] item-name [ args ... ]
    
      'args' are optional 'op create item \"Secure Note\"' arguments
      which could contain '--vault=XXX' or '--tag=FOOBAR';
      but don't use '--title=XXX'
    
      returns 0 if the item has been created or updated, otherwise non-zero
    
      option -i, -I, or --stdin  means take item contents from stdin;
      option -c, -C, or --create means only create when item doesn't exist;
      option -u, -U, or --update means only update when item does exist;
      with neither --create nor --update, decide which to do based upon existence;
      --create and --update cannot both appear
    '''.format(prog))
        return -2
    
    if __name__ == '__main__':
        sys.exit(main())
    
  • HippoMan
    HippoMan
    Community Member
    edited May 2018

    op-rm-note - delete a Secure Note

    usage: op-rm-note item [ optional 'op delete item' arguments ]
    
      returns 0 if the item is deleted, otherwise non-zero
    
      optional arguments could contain '--vault=XXX'
    

    Python code ...

    #!/usr/bin/python3
    
    import sys
    sys.dont_write_bytecode = True
    
    import os
    
    sys.path.insert(0, '/usr/local/etc/python')
    from u import invoke
    
    prog       = os.path.basename(sys.argv[0])
    op         = '/usr/local/bin/op'
    itemexists = '/usr/local/bin/op-item-exists'
    
    def main():
    
        if len(sys.argv) < 2:
            return usage()
    
        name   = sys.argv[1]
        base   = os.path.basename(name)
        params = sys.argv[2:]
    
        rc, o, e = invoke([ itemexists, base ] + params)
        if rc:
            return rc
    
        uuid = o.rstrip()
    
        rc, o, e = invoke([ op, 'delete', 'item', uuid ] + params)
        return rc
    
    def usage(msg=None):
        if msg:
            print('\n{}'.format(msg))
        print('''
    usage: {} item [ optional 'op delete item' arguments ]
    
      returns 0 if the item is deleted, otherwise non-zero
    
      optional arguments could contain '--vault=XXX'
    '''.format(prog))
        return -2
    
    if __name__ == '__main__':
        sys.exit(main())
    
  • HippoMan
    HippoMan
    Community Member
    edited May 2018

    op-create-note - thin shell wrapper around op-put-note for creating Secure Notes ...

    #!/bin/zsh -f
    prog=${0##*/}
    exec -a ${prog} /usr/local/bin/op-put-note --create "${@}"
    exit -1
    

    op-update-note - thin shell wrapper around op-put-note for updating Secure Notes ...

    #!/bin/zsh -f
    prog=${0##*/}
    exec -a ${prog} /usr/local/bin/op-put-note --update "${@}"
    exit -1
    

    Note: exec -a ${prog} is a zsh construct to force argv[0] to be set in the executed program.

  • HippoMan
    HippoMan
    Community Member

    Note that these python programs use some functions from a module which I called u. Here's the __init__.py code for this u module, wherein these functions are defined ...

    #!/usr/bin/python3
    
    import sys
    sys.dont_write_bytecode = True
    
    import os
    import warnings
    import unidecode
    import subprocess
    
    def utf8decode(text, rstripnull=True):
        if not text:
            return ''
        if type(text) is bytes:
            if rstripnull:
                try:
                    text = text.rstrip(b'\0')
                    if not text:
                        return ''
                except:
                    pass
            try:
                text = text.decode('utf8')
            except:
                text = text.decode('utf8', errors='replace')
            if not text:
                return ''
        try:
            with warnings.catch_warnings() as w:
                warnings.simplefilter('ignore')
                text = unidecode.unidecode(text)
        except:
            text = ''
        return text
    
    def _communicate(subproc, inputstr=None):
        if inputstr is None:
            return subproc.communicate()
        if not inputstr:
            inputstr = ''
        if sys.version_info >= (3,):
            return subproc.communicate(inputstr.encode())
        else:
            return subproc.communicate(inputstr)
    
    def invoke(cmd, input=None, evars=None):
        e = os.environ.copy()
        if evars:
            for k,v in evars.items():
                e[k] = v
        p = subprocess.Popen([str(x) for x in cmd], shell=False, bufsize=0, env=e, close_fds=True,
                             stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        try:
            out, err = _communicate(p, inputstr=input)
        except Exception as e:
            out = ''
            err = ''
        rc = p.returncode
        return (rc, utf8decode(out), utf8decode(err))
    
  • This is incredible. You should throw that stuff onto a github repo to make it easily sharable with others.

    Wow you're awesome. :)

    Rick

  • HippoMan
    HippoMan
    Community Member

    Thank you!

    Yes, I plan to github-ize it soon.

  • AGAlumB
    AGAlumB
    1Password Alumni

    Please do! SO great! Secure Notes are my FAVOURITE. :chuffed:

  • cohix
    cohix
    1Password Alumni

    @HippoMan This is damn fantastic. If you create a GitHub repo for this, I would love to fork and contribute comparable Go wrappers. It would be amazing to start a community-driven repository of scripts.

    Seriously, thank you for all the hard work you've been doing over the past few weeks.

  • HippoMan
    HippoMan
    Community Member

    Thank you!

    I started a new job and have less spare time now, so it might be a couple weeks before I can get the github repo set up. But I will definitely do it.

  • HippoMan
    HippoMan
    Community Member
    edited June 2018

    Also, here are some proposed enhancements of the CLI code base which could make these python wrapper efforts more efficient:

    • The ability in the CLI to query items (documents, secure notes, etc.) in the Trash and also to be able to delete them from the Trash and restore them from the Trash to non-Trash locations.
    • Return codes from op list items which give us information about how many items are found, so we don't have to parse error strings.
    • The ability in the CLI to rename items (documents, secure notes, etc.)
    • The ability in the CLI to update items (documents, secure notes, etc.)
    • Some sort of op status command which will return information about whether the current session is valid ... via return codes, not simply by means of error strings which would then need to be parsed.

    With these capabilities (especially the first one), the CLI wrappers will be a lot more efficient. And if the first four of these are implemented, the wrappers might not even be needed any more.

    And as I mentioned elsewhere, I'm willing to sign a non-disclosure agreement and work on making these changes in a copy of the official code base. I'm learning go, and I'm sure I could implement these enhancements.

  • The ability in the CLI to query items (documents, secure notes, etc.) in the Trash

    Does the --include-trash flag on op get item and op list items not do what you're looking for? Can you help me understand what you're trying to do?

    to be able to delete them from the Trash

    The next release should have a op delete trash command to empty the trash. Hopefully this will do the trick for you.

    restore them from the Trash to non-Trash locations.

    Interesting. Makes sense. Filed as issue 447 in our tracker so that we can consider that.

    Return codes from op list items which give us information about how many items are found, so we don't have to parse error strings.

    I think you might mean op get item here? Counting items from op list items should be pretty easy as it's an array of objects. Error codes in general need some rethinking in our tool though. We seem to want them to mean a few different things and there isn't enough consistency.

    The ability in the CLI to [rename, update] items (documents, secure notes, etc.)

    We'll get there. I suspect both renaming and general updating will be done via the same command (op edit item) when we roll that out.

    Some sort of op status command

    That'd be good, but it feels like a bit of a hack to me. I'd much rather we build something where you didn't need to worry about the status of your session. If the session dies for whatever reason the client should be able to re-negotiate a new one with the server. Something like eval $(op signin agilebits --longlived) and from then on you can assume your session is always available until you close that terminal. Otherwise you'd be stuck running op status between every command to possibly renegotiate a new session if needed (the server reserves the right to invalidate a session for any reason it sees fit).

    Rick

  • HippoMan
    HippoMan
    Community Member
    edited June 2018

    Sorry for taking so long to reply. I've been busy at my new job.

    First of all, THANK YOU (!!!) for op delete trash! This plus --include-trash will allow me to greatly simplify the python wrappers, and it makes the CLI much more usable.

    Yes, I meant op get item. If that call fails, the only way we can know that the failure is due to there being multiple matched items is to parse the error messages. It's highly inefficient to always have to do op list items in conjunction with every op get item failure, just to be able to find out whether the failure was due to a multiple-item match. It would be cleaner and more efficient if we could know of this multiple-match case via a simple op get item return code.

    Also, when using --include-trash, is it possible to distinguish between Trash items and other items? I seem to recall that all returned items have a trash or similar JSON attribute set to "N", even for items in the Trash. Perhaps I'm wrong about this, however (I can't test this at the moment, because I'm away from my desktop machine and posting via the mobile app).

    Anyway, thank you for these enhancements, and especially for the upcoming op delete trash!

  • It would be cleaner and more efficient if we could know of this multiple-match case via a simple op get item return code

    Yes. We need to get that error to you somehow.

    Also, when using --include-trash, is it possible to distinguish between Trash items and other items? I seem to recall that all returned items have a trash or similar JSON attribute set to "N", even for items in the Trash.

    I'm not quite sure I get what you're asking here. All items will have a trashed attribute which should be either Y or N depending on whether it's in the trash or not.

    Anyway, thank you for these enhancements, and especially for the upcoming op delete trash!

    You're welcome. Hopefully we can get that release out soon. We've been busy working on the SCIM bridge lately. We need more hours in a day!

    Rick

  • HippoMan
    HippoMan
    Community Member

    I will double-check when I get home, but I seem to recall that trashed is always returned as "N" in all cases, even for items in the Trash. Or perhaps I'm thinking of an older CLI version. I'll report back.

  • AGAlumB
    AGAlumB
    1Password Alumni

    If so, that definitely sounds like a bug. Let us know what you find!

  • HippoMan
    HippoMan
    Community Member

    I have checked, and trashed now seems to indeed be set correctly. I believe I was thinking about an earlier CLI version, and I'm sorry for the false alarm.

  • AGAlumB
    AGAlumB
    1Password Alumni

    No worries! Thanks for checking. I don't recall an issue with that, but a lot has happened this past year too, so you may be right. :)

This discussion has been closed.