Using MrC's converter suite moving from LastPass... -a / --addfields not working?

Does this functionality still work?

The import file doesn't create a custom field and the Field/Value pairs are just placed into the Notes section as if it wasn't specified in the original conversion command.

I'm using what I think is the latest version - 2019-05-22-2109

I've tried from both Windows and macOS but still the same outcome. I've also tried a new data export in case that was the case but still it all ends up in the notes section.


1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

Comments

  • Greg
    Greg
    1Password Alumni

    Hi @TobyKershaw,

    Thank you for getting in touch!

    Conversion process has a lot of moving parts, so I am not sure what is happening here. Am I right to understand that our standard migration process doesn't suit your needs and you want to use @MrC 's converter suite to customize your import? Please confirm.

    If you haven't seen our migration guide, please check it out:

    Move your data from LastPass to 1Password

    Thank you!

    ++
    Greg

  • TobyKershaw
    TobyKershaw
    Community Member

    Hi Greg,
    Yes the native import tools don't seem to offer the ability to convert custom fields into fields?

    Would also be great if there was some means of mapping custom templates to either stock or custom 1Password templates.

  • MrC
    MrC
    Volunteer Moderator

    @TobyKershaw ,

    I just checked the option and it seems to be working for me:

    $ perl convert.pl lastpass _TestFiles/lastpass/lastpass_mac.csv -v --addfields
    Imported 20 items
    Exported 1 wireless item
    Exported 1 creditcard item
    Exported 3 membership items
    Exported 1 database item
    Exported 2 note items
    Exported 1 passport item
    Exported 1 email item
    Exported 3 login items
    Exported 1 driverslicense item
    Exported 1 software item
    Exported 2 server items
    Exported 2 bankacct items
    Exported 1 socialsecurity item
    Exported 20 total items
    You may now import the file /Users/MrC/Desktop/1P_import.1pif into 1Password

    The video shows the same entry with and without --addfields.

    Can you provide any more clues?

  • TobyKershaw
    TobyKershaw
    Community Member

    Hi @MrC,

    I've just made an entry, exported from LastPass, chopped out all other entries and was left with two lines in a CSV to import - the header and the one entry. I've then ran "perl convert.pl lastpass ..\win_test_import.csv -v --addfields" in PowerShell (and again in CMD).
    I get good feedback from the verbose logging:

    "Imported 1 item
    Exported 1 note item
    Exported 1 total item
    You may now import the file C:\Users\TobyKershaw\Desktop\1P_import.1pif into 1Password"

    and upon import I get this:

    I am guessing that as it's not detecting the note type correctly that something has gone wrong with my export?

  • TobyKershaw
    TobyKershaw
    Community Member

    Just did a conversion of a raw export from Firefox (went back a re-read the instructions to rule that out) and noticed that of the 4623 entries, they've either been detected as login items or notes...no other types despite there being a mixture of many:

    Imported 4623 items
    Exported 2468 note items
    Exported 2155 login items
    Exported 4623 total items

  • MrC
    MrC
    Volunteer Moderator

    @TobyKershaw ,

    Let's focus on one type of export at a time.

    Unless you fully understand the export format of the Lastpass export, you should not manually edit it to work with the converter. It is not a simple CSV file.

    If you are trying to see what is happening during conversion, use -d to debug:

    $ perl convert.pl lastpass _TestFiles/lastpass/lastpass_mac.csv -vd  --imp healthinsurance
    
    do_import           : ROW: 9
    do_import           :       type determined as 'healthinsurance'
    normalize_card_data : field: Company
    normalize_card_data : field: Company Phone
    normalize_card_data : field: Policy Type
    normalize_card_data : field: Policy Number
    normalize_card_data : field: Group ID
    normalize_card_data : field: Member Name
    normalize_card_data : field: Member ID
    normalize_card_data : field: Physician Name
    normalize_card_data : field: Physician Phone
    normalize_card_data : field: Physician Address
    normalize_card_data : field: Co-pay
    print_record        : title: Note: Health Insurance
                          tags:  Secure Notes
                          key(copay): Co-pay = $20
                          key(physicianaddr): Physician Address = The Hospital
                          key(physicianphone): Physician Phone = 555-2222
                          key(physician): Physician Name = Doc McQuack
                          key(membership_no): Member ID = 55555
                          key(member_name): Member Name = Joe User
                          key(grpid): Group ID = 9999
                          key(polid): Policy Number = 1234
                          key(poltype): Policy Type = group
                          key(phone): Company Phone = 555-1212
                          key(org_name): Company = HealthCo
                          notes: Notes:some doc notes<CR>
    
  • MrC
    MrC
    Volunteer Moderator

    @TobyKershaw ,

    The goal is to find the reason why just one of the record's is being mis-categorized. This usually leads to discovering the reason for all of them.

    If you do the export and conversion on a Mac, you'll find the command line tools there easier to use to help identify this. Are you versed enough in command line tools to do this?

  • TobyKershaw
    TobyKershaw
    Community Member

    @MrC Totally understand that its not an ordinary CSV but even with a raw untouched export from Firefox or Chrome I don't get any matches besides Login Items and Notes across the 4600 items in there.

    With earlier testing I got the same results from a macOS device ...but I was trying with the same manipulated CSV file. I'll try with a raw export and will report back the results.

    Thanks for the pointers!

  • MrC
    MrC
    Volunteer Moderator
    edited June 2019

    @TobyKershaw ,

    Most of the categories in Lastpass have their field/value pairs packed into the notes (extra) area of the item. The first part of that data indicates the category, and then it is followed by the category's field data. When this area contains the string "http://sn", it gets category-detected and the field/value pairs are pulled. When it has something other than this string, it is a Login category.

  • TobyKershaw
    TobyKershaw
    Community Member

    @MrC Having re-exported on a macOS based device (used Safari, saves as UTF-8 txt file, etc.) and there are profiles being detected now:

    It must be due to the way the csv/txt file is being generated in my Chrome/Firefox on the Windows based machine.

    When it comes to editing the export, is it even possible? Or should I look to spin up a dedicated LastPass user with limited access to specific shares and export like that?

  • MrC
    MrC
    Volunteer Moderator

    @TobyKershaw ,

    I know what the issue is now. The Lastpass export via extension used to be consistent with how it exported the data. It now seems to vary with browser and OS, particularly with the newline sequence. It also seems to add useless blank lines to the export data on the web page it opens, so these appear in the text file when pasting the copied data.

    Let me correct the converter to deal with these issues...

  • MrC
    MrC
    Volunteer Moderator
    edited June 2019

    @TobyKershaw ,

    I've updated the converter suite to hopefully handle the issues you've encountered on Windows:

    2019-06-14 -----------------------------------------------------------------------------------------

    • Fix [lastpass]
      Handle the various forms of BOM, newline endings, and saved file formats from more recent versions
      of LastPass via the browser or the extension.

    Please give it a try to see if it works for you (either platform should be fine).

    To answer your question above about editing the saved LastPass data, yes, you can edit it. Just be sure you use a plain text editor, and understand the boundaries of the virtual "rows" of data.

  • TobyKershaw
    TobyKershaw
    Community Member

    @MrC The change in version has resolved the detection of categories but -a / --addfields is still not functioning for me? All data value pairs are just dumped into the notes section - this is true of both Windows and macOS generated .1pif file. Could there still be an issue with data I'm exporting from LastPass?

  • TobyKershaw
    TobyKershaw
    Community Member

    @MrC OK don't worry, I figured it out.

    I was hoping the convertor would be able to handle custom templates and just create fields for the label:value pairs.

  • MrC
    MrC
    Volunteer Moderator

    @TobyKershaw ,

    Sorry for the delay - Comcast is having fits and starts the past 24 hours.

    Everything in the converter is driven by the Category / Field table at the beginning of each converter. This defines the input category, and maps it to a native 1Password category (and possibly splits the record into multiple output categories, to allow 1Password to work best).

    Any input "key" that is not in that table is considered to be a simple key/value field, and gets targeted to the Notes area. The --addfields globally tells the 1PIF generator to add these key/value fields to a custom section in the 1Password record (the section is named "Original Fields", and is defined in the converter module Utils/PIF.pm).

    The table of categories / fields at the beginning of each converter module can be customized so that it recognizes user's custom fields. Obviously that's not something I could add generally, since each user has their own ideas about representing their data. But you can modify those entries, and I'm happy to guide you there. You can also add new custom import categories / fields to match your data. Some exports will be limited here, due to the inadequacies of their export data and format.

    Why don't we take this offline - you can reach out to me via email (top of the convert.pl script) and I can explain further with examples of how this works. We can summarize here when done.

  • MrC
    MrC
    Volunteer Moderator

    We're handling this offline via email. We'll report back here when done.

    @TobyKershaw

  • MrC
    MrC
    Volunteer Moderator
    edited June 2019

    @TobyKershaw and I worked offline to resolve the questions and issues.

    The converter suite has been updated with the following relevant changes:

    • Fix [lastpass]
      Handle Lastpass' grouping (folder) hierarchy character to work with 1Password tags.

    • New [lastpass]
      The converter will now split all key:value pairs inside non-Generic notes types. This makes the
      fields available for the --addfields option.

    • New [lastpass]
      The %card_field_specs table accepts 'textname' values as either strings or regular expressions.
      This allows matching multiple template names (their names are exported in the form Custom_###
      where ### is a sequence of digits).

This discussion has been closed.