MrC's Convert to 1Password Utility (mrc-converter-suite)

14647485052

Comments

  • MrC
    MrC
    Volunteer Moderator
    edited January 2019

    @Enterhaken @goelli

    Looks like a module that I use is not present in Strawberry Perl. I'll add it to the module list to include.

    Easy fix. For you, in the portableshell.bat command shell window, type:

    cpan install -f Win32::Unicode::File
    

    and hit Enter. It will install the module, and you should be able to try the converter again.

  • goelli
    goelli
    Community Member

    Thank you very much. I tried to install the modul, but it failed because of a test. After installing it with force option like you suggested, your script runs fine. It decoded all my Firefox passwords and stored it in the export file.

    This saved so much time :-)

  • MrC
    MrC
    Volunteer Moderator
    edited January 2019

    @goelli

    Fantastic!

    Yes, the module currently has two minor printing bugs in tests, which causes two of them too fail. They don't affect my usage of it. I'm working now on the best method to deal with this.

  • mkhoury
    mkhoury
    Community Member
    edited January 2019

    I can't copy anything from iCloud Keychain to 'local-cloud' without getting this error, "An error has occurred. Unable to add an item to the current keychain. One or more parameters passed to a function were not valid." This is so frustrating, I just want my passwords OUT of iCloud Keychain!

  • MrC
    MrC
    Volunteer Moderator

    @mkhoury ,

    Please see this post. Make sure you have version 1.11 of the converter suite. The scripts are included with the archive now, and not in the Testing Bits area.

    The README and converter docs are being updated to explain more about the issues with copying iCloud keychain in macOS versions Sierra and beyond. A preview:

  • mkhoury
    mkhoury
    Community Member

    Thanks! That did the trick!

  • MrC
    MrC
    Volunteer Moderator
    edited January 2019

    I've posted a significant update to the converter suite.

    • The convert_to-1p4 naming is retired - the suite is now called the mrc-converter-suite.
    • The Stable Bits / Testing BIts areas are retired. Instead, I will update the single, dated archive when I make changes.
    • The README.pdf is retired, replaced by an much better, easy flowing README.html.
    • All documentation has been re-written and much of it auto-generated.
    • For Windows users, the installer version of Strawberry Perl is now prescribed instead of the portable version.
    • See the Changes.txt file for a list of changes.

    I've tested every step, reread the docs a dozen times, but most certainly I've overlooked something.

  • mkhoury
    mkhoury
    Community Member
    edited January 2019

    I'm using the converter, since importing from CSV is mysteriously not working anymore. The converter outputs the .1pif file but everything imports as NOTES and not Logins.

    Update: importing the same CSV file via the website works.

  • MrC
    MrC
    Volunteer Moderator
    edited January 2019

    @mkhoury ,

    I'm not seeing any issues here:

    $ cat  _TestFiles/CSV/login-utf8.csv
    "title","login username","login password","login url","notes","tags","cust col 1","created","modified"
    "Joe's Login","Joe Blow","supersecret","http://www.example.com","what notes?","TAG1","cust val 1","1522630494","1522604494"
    "Sally's Login","Sally Smith","supersecret","http://xxx.example.com",Üñîçø∂é,"TAG2","cust val 1","1522630494d","1522604494x"
    
    $ perl convert.pl csv -v _TestFiles/CSV/login-utf8-win.csv
    Imported 2 items
    Exported 2 login items
    Exported 2 total items
    You may now import the file /Users/MrC/Desktop/1P_import.1pif into 1Password
    
    $ perl convert.pl csv -v _TestFiles/CSV/login-utf8.csv
    Imported 2 items
    Exported 2 login items
    Exported 2 total items
    You may now import the file /Users/MrC/Desktop/1P_import.1pif into 1Password
    
    $ perl convert.pl csv -vd _TestFiles/CSV/login-utf8.csv
    main                : Runninng script from '/Users/MrC/Documents/Coding/mrc-converter-suite'
    main                : Command Line: csv -vd _TestFiles/CSV/login-utf8.csv
    main                : Output file: /Users/MrC/Desktop/1P_import.1pif
    print_fileinfo      : Input file info: "_TestFiles/CSV/login-utf8.csv"
    print_fileinfo      :   size: 360
    print_fileinfo      :   kind: UTF-8 Unicode text
    print_fileinfo      :   mime: text/plain; charset=utf-8
    find_card_type      :       type detected as 'login'
    do_import           : ROW: 1
    normalize_card_data : field: login username
    normalize_card_data : field: login url
    normalize_card_data : field: cust col 1
    normalize_card_data :   pushed to notes: cust col 1: cust val 1
    normalize_card_data : field: login password
    print_record        : title: Joe's Login
                          tags:  TAG1
                          key(password): login password = supersecret
                          key(url): login url = http://www.example.com
                          key(username): login username = Joe Blow
                          notes: cust col 1: cust val 1<CR><CR>what notes?
    do_import           : ROW: 2
    do_import           : Invalid modified epoch date: 1522604494x
    do_import           : Invalid created epoch date: 1522630494d
    normalize_card_data : field: modified
    normalize_card_data : field: created
    normalize_card_data : field: login password
    normalize_card_data : field: login url
    normalize_card_data : field: cust col 1
    normalize_card_data :   pushed to notes: cust col 1: cust val 1
    normalize_card_data : field: login username
    print_record        : title: Sally's Login
                          tags:  TAG2
                          key(username): login username = Sally Smith
                          key(url): login url = http://xxx.example.com
                          key(password): login password = supersecret
                          key(created): created = 1522630494d
                          key(modified): modified = 1522604494x
                          notes: cust col 1: cust val 1<CR><CR>Üñîçø∂é
    Imported 2 items
    create_pif_record   : Title: Joe's Login
    create_pif_record   :   key test(username): Joe Blow
    create_pif_record   :   key test(url): http://www.example.com
    create_pif_record   :   key test(password): supersecret
    create_pif_record   :   notes: cust col 1: cust val 1<CR><CR>what notes?
    create_pif_record   :   tags: TAG1
    create_pif_record   : Title: Sally's Login
    create_pif_record   :   key test(modified), Not found
    create_pif_record   :   key test(created), Not found
    create_pif_record   :   key test(password): supersecret
    create_pif_record   :   key test(url): http://xxx.example.com
    create_pif_record   :   key test(username): Sally Smith
    create_pif_record   :   notes: cust col 1: cust val 1<CR><CR>Üñîçø∂é
    create_pif_record   :   tags: TAG2
    create_pif_record   :  *unmapped card field pushed to notes: modified
    create_pif_record   :  *unmapped card field pushed to notes: created
    Exported 2 login items
    Exported 2 total items
    You may now import the file /Users/MrC/Desktop/1P_import.1pif into 1Password
    

    What platform?
    Are you using the new mrc-converter-suite package?
    Do you have the correct column labels?

  • mkhoury
    mkhoury
    Community Member

    Used the conversion script to take password out of Safari and into CSV (like I've done on two other computers now). Then I tried to import the CSV into the 1Password macOS app. It kept throwing an error. Tried the conversion script and that wouldn't convert anything but the stuff in the CSV into "notes" (didn't want that). Thought about trying the CSV importer via the web, that worked like a charm.

  • MrC
    MrC
    Volunteer Moderator
    edited January 2019

    Aha!

    @mkhoury - I found a bug and fixed it. Can you give the update I just posted a try?

    2019-01-25 -----------------------------------------------------------------------------------------

    • Fix [csv]
      Fix issue mishandling special columns that were not present in the CSV (e.g. tags, notes, created,
      modified).
  • mkhoury
    mkhoury
    Community Member

    That did the trick!

  • djshockd
    djshockd
    Community Member
    edited January 2019

    Hi @MrC.

    Thank you for writing such a wonderful and well-documented conversion tool!

    I get the error message below when trying to convert from a KeePass 2 XML file. For context, the process to create the file was as follows:

    1. Windows XP KeePass 1.x file (.kdb) imported into Windows KeePass 2.x (.kdbx)
    2. Exported Windows KeePass 2.x (.kdbx) to KeePass 2 XML file
    3. XML file is saved to desktop and shared between Windows XP VM and Mac via Parallels
    4. Converter is run on Mac using macOSConvertHelper to generate command in Terminal

    Error message:

    Uncaught exception from user code:
    Error parsing time at /System/Library/Perl/5.18/darwin-thread-multi-2level/Time/Piece.pm line 469.
    Time::Piece::strptime('Time::Piece', '0002-11-30T08:00:00Z', '%Y-%m-%dT%H:%M:%SZ') called at /Users/roopesh/Desktop/mrc-converter-suite/./Converters/Keepass2.pm line 278
    Converters::Keepass2::parse_date_string('0002-11-30T08:00:00Z') called at /Users/roopesh/Desktop/mrc-converter-suite/./Converters/Keepass2.pm line 286
    Converters::Keepass2::date2epoch('0002-11-30T08:00:00Z') called at /Users/roopesh/Desktop/mrc-converter-suite/./Converters/Keepass2.pm line 124
    Converters::Keepass2::do_import('/Users/roopesh/Desktop/pm_export.txt', undef) called at convert.pl line 169

  • MrC
    MrC
    Volunteer Moderator

    Hi @djshockd ,

    The issue is that a date in your file is unexpectedly bogus. The date string is:

    0002-11-30T08:00:00Z
    

    and clearly 0002 is not a valid (modern age) year. I can code around that. I'll post an update in a few minutes.

  • MrC
    MrC
    Volunteer Moderator

    @djshockd ,

    Update posted.

    Thanks for you kind words!

  • djshockd
    djshockd
    Community Member
    edited January 2019

    @MrC,

    I figured, and tried searching within KeePass to find the problematic item but it doesn’t return any results. Perhaps the export process modifies the date formats. I suppose I could regenerate the XML and try searching directly. But since you have an update I’ll give that a try!

    Do you accept donations anywhere for your efforts?

  • MrC
    MrC
    Volunteer Moderator
    edited January 2019

    @djshockd ,

    It is a curious one, and I've never encountered such an issue w/KeePass 2, so have no idea how that date was derived. You can turn on debugging if you are curious to find it. You'd look for lines such as:

    get_entrydata_from_entry:       key: LastModificationTime: '2015-04-30T03:14:20Z'
    get_entrydata_from_entry:       key: CreationTime: '2015-04-30T03:12:20Z'
    

    Debugging is enabled with -d or --debug :

    perl convert.pl keepass2 -dv TestFiles/keepass2.xml
    

    But its no big deal, as testing for the failed date parsing is something that I should do anyway. One simple reason I had not done this - I want to get feedback like yours, since I can't create all the possible permutations of user data (I do try to test all the edge cases I can think of).

    Your compliments and offer have been a sufficient donation to me.

  • d2theustin
    d2theustin
    Community Member

    @MrC if I quit the apple script before it completes, are the passwords that it has captured saved somewhere? I want to ensure they get deleted.. I decided to go a different route than copying plain text.

  • djshockd
    djshockd
    Community Member

    @MrC , just wanted to give the feedback that the update worked. Thanks for the fix!

  • MrC
    MrC
    Volunteer Moderator
    edited January 2019

    @djshockd - That's great news. Enjoy 1Password!

    @d2theustin - Which script - there are 3 AppleScripts - the macOSConvertHelper and two Get_Safari##_Passwords scripts. Just so I'm clear.

    The two Get_Safari... scripts do not write out their contents until all rows have been processed. The internally collected data dies with script termination.

  • Nonylus
    Nonylus
    Community Member

    Hi MRC,

    First, like most of us, a HUGE immense thank you for your work on these conversion tools.

    Here's the issue I'm encountering trying to run the convert.pl for chrome

    $ perl convert.pl -v chrome
    Error: failed to load converter module 'chrome'
    Can't locate PBKDF2/Tiny.pm in @INC (you may need to install the PBKDF2::Tiny module) (@INC contains: /Users/username/Desktop/mrc-converter-suite/. /Users/username/Desktop/mrc-converter-suite/lib /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.2 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extras/5.18/darwin-thread-multi-2level /System/Library/Perl/Extras/5.18 .) at /System/Library/Perl/5.18/if.pm line 13.
    BEGIN failed--compilation aborted at /Users/debbienordstrom/Desktop/mrc-converter-suite/./Converters/Chrome.pm line 23.
    Compilation failed in require at convert.pl line 53.
    
    Usage: convert.pl <options> chrome <export file>
    
    Options:
    
    Select one of the converters above and add it to the command line to see more
    complete options.  Example:
    
        perl convert.pl ewallet --help
    

    I've tried using /usr/bin/perl5.16 or /usr/bin/perl5.16 convert.pl -v chrome with the same results.
    Similar issue with csv option by the way (I was able to work around that one by using a different computer running Mojave to convert the .csv file, a Safari password export)

    I can't use that same work around with Chrome as it doesn't use an export file that aI can copy to another computer.
    Any ideas?

    Merci beaucoup!

    System Info:
    Downloaded mrc-convert-suite Today (Latest I imagine)
    macOS 10.11.6
    Chrome 71.0.3578.98 Official Build 64bit

  • Nonylus
    Nonylus
    Community Member
    edited February 2019

    Hi MRC,

    First, like most of us, a HUGE immense thank you for your work on these conversion tools.

    Here's the issue I'm encountering trying to run the convert.pl for chrome

    $ perl convert.pl -v chrome
    Error: failed to load converter module 'chrome'
    Can't locate PBKDF2/Tiny.pm in @INC (you may need to install the PBKDF2::Tiny module) (@INC contains: /Users/username/Desktop/mrc-converter-suite/. /Users/username/Desktop/mrc-converter-suite/lib /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.2 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extras/5.18/darwin-thread-multi-2level /System/Library/Perl/Extras/5.18 .) at /System/Library/Perl/5.18/if.pm line 13.
    BEGIN failed--compilation aborted at /Users/username/Desktop/mrc-converter-suite/./Converters/Chrome.pm line 23.
    Compilation failed in require at convert.pl line 53.
    
    Usage: convert.pl <options> chrome <export file>
    
    Options:
    
    Select one of the converters above and add it to the command line to see more
    complete options.  Example:
    
        perl convert.pl ewallet --help
    

    I've tried using /usr/bin/perl5.16 or /usr/bin/perl5.16 convert.pl -v chrome with the same results.
    Similar issue with csv option by the way (I was able to work around that one by using a different computer running Mojave to convert the .csv file, a Safari password export)

    I can't use that same work around with Chrome as it doesn't use an export file that aI can copy to another computer.
    Any ideas?

    Thanks amillio

    System Info
    Downloaded mrc-convert-suite Today (Latest I imagine)
    macOS 10.11.6
    Chrome 71.0.3578.98 Official Build 64bit

  • markaceto
    markaceto
    Community Member

    Hi MRC,

    When running macOSConvertHelper to convert the Safari passwords CSV, I keep getting the error in the attached screenshot.

    I'm running 10.13.6 on an iMac Pro.

  • markaceto
    markaceto
    Community Member
    edited February 2019

    Importing the CSV file (vs converting it) seemed to work fine.

    However, when reviewing duplicate passwords, one of the first items that I updated brought in 4 additional sites with the same domain. The first password I changed updated 2 domains. The next password I changed updated 1 domain, and I still 2 domains to go buried in the notes of the imported item, resulting in a bunch of copy/pasting and logging in/out of multiple browser tabs. Therefore, I would definitely prefer to have each (Safari) shared domain broken out as 5 separate items (in this case), as the 1:1 changes would be more streamlined, and less confusing to track.

  • MrC
    MrC
    Volunteer Moderator
    edited February 2019

    Hi @markaceto ,

    Let me see if I can replicate the issue you see on 10.13 with macOSConvertHelper. You can just use the command line for now if necessary - the macOSConvertHelper job is to just build it for you anyway.

    RE: the multiple web-host names that can come out of Safari when it groups like-items, it is an annoying Safari behavior. I don't think I want to have the AppleScript code split records for several reasons. Rather, what I'd prefer to do is create a special CSV column and place those items in that column. Then program the csv converter to know about that special column, and based on an option, let the converter do the record-splitting. This is a more generic solution that could be applicable to other CSV exports, and frankly, the converter suite's scripting language is much agile and easier to work with than AppleScript (which is kind of like teaching a child to ride a bike by instructing, now flex this muscle slightly, elongate this limb a little, lean to your left side, push, turn the handlebars 2 degrees right, lean...)

    How does that sound?

    Edit: I believe the issue with the script being unable to a Terminal window is caused if you don't Allow Script Editor access to Terminal. macOS will prompt you to Allow access. Perhaps that dialog was not noticed, or cancelled.

  • markaceto
    markaceto
    Community Member
    edited February 2019

    Thanks for the quick reply!

    Unless I'm mistaken, the conversion is an extra step because I can just import the CSV directly into 1p 6.8.9 (not sure about v7).

    RE: the multiple web-host names that can come out of Safari... programming in AppleScript sounds like a nightmare, so your proposed solutions sounds good. I'm concerned about (my own) user error when I get lost in the minutia, so whatever is the most bulletproof solution works for me :)

    BTW:
    1. Scrolling through the list of passwords in Safari prefs, I have roughly 60 grouped items (ranging from 1-4 other instances each) out of 340 "subtotal" so I actually have upwards of 400 "grand total". I would definitely screw up a few of those, even when working in small batches.
    2. I also have the issue of a blank website reappearing from iCloud sync, and I also have to quad-click it to open the detail sheet. The first double-click does nothing. The second double-click opens the sheet.

    I'm really excited about getting out of Safari/iCloud/Keychain, and into a more user-serviceable system!

  • MrC
    MrC
    Volunteer Moderator

    @markaceto ,

    You're welcome.

    Yes, you can import directly. The CSV generated by the applescript is very simple - there's not much data there - just a few cells / row. The converter proper provides more abilities, but they may be inconsequential for your needs. I just don't want to spend much more time working on both of the AppleScripts (which seem to require updates each macOS / Safari release), and the script flow is based on one CSV row at a time - and I certainly don't want to start adding options processing (some users will want the URLs grouped).

    Since 1Password has the ability to handle multiple URLs per Login record, the right want to deal with this is for the csv converter to place them into those supplementary URL areas rather than notes. This way, I can add an option (like --addurls, I'll think about a good name), and make it user's choice. With this, inside of 1Password you have one record, but you can click on any of the links. I could also provide the ability to add the URLs to Notes, or split the records, so you can choose how to handle them.

    Eh, gads. I've almost never saved passwords in browsers. I didn't know about Safari grouping items until someone reported the issue, and I worked very hard to figure out how they those entries were created. Never occurred to me Safari would group them by itself automatically (because its basically the wrong approach, and error prone). Nor did I know how to create blank entries, but eventually figured it out (bug in older macOS'). And the blank entries not opening without effort, or them even being stored at all, is clearly a Safara / iCloud bug. The applescript tries to work around this issue. Is it failing for you?

  • MrC
    MrC
    Volunteer Moderator

    @markaceto

    Sorry, which version of Safari are you using on 10.13 where you ran the script?

  • markaceto
    markaceto
    Community Member
    edited February 2019

    @mrc,

    If Apple hadn't broken iCloud Keychain export, I'd be using that for sure. My plan is to get all the passwords out of Safari, change the duplicate/weak/vulnerable ones, and then delete them from Safari. That should also purge them from iCloud Keychain, leaving a significantly more manageable batch for me to start pulling into 1p. My goal is to strip everything but system level stuff out of the Keychain. Process of elimination...

    I'm using Safari 12.0.3

  • MrC
    MrC
    Volunteer Moderator

    @markaceto ,

    Yup, I get it.

    I need to create a VM environment on 10.13 to test out the issue w/Safari 12. I have 11 on my VM currently. I'll see what I can find.

This discussion has been closed.