DataVault items converted as Secure Notes

I converted from DataVault to 1Password following the converter instructions. 17 of my items converted as Logins and ~450 converted as secure notes. I looked into the CSV and it is unclear the difference in the file.


1Password Version: 6.2.333d
Extension Version: Not Provided
OS Version: W 10
Sync Type: Not Provided
Referrer: forum-search:DataVault Imports are Secure Notes

Comments

  • MrC
    MrC
    Volunteer Moderator

    Hi @mstasim ,

    Have you reviewed all the caveats about the DataVault export in the README.pdf ?

    The DataVault export is very weak.

    Can you identify which record types are being mis-categorized?

  • Greg
    Greg
    1Password Alumni

    Hello @mstasim,

    Thank you for sharing your experience! I am sorry you bumped into troubles while migrating to 1Password. :(

    The import process from various applications may be tricky, but the converter tool was developed by MrC, so his questions are always on point. Please share this additional info, so we could get the ball rolling. :+1: Thank you in advance!

    Cheers,
    Greg

  • mstasim
    mstasim
    Community Member

    I read all the PDF data. The main one is logins. Seems like the converter gets username password, but everything beyond that sets dumped into notes. No URL or Secret Word conversion. On many of the logins, it pulled even my username and password into the secure notes...

  • MrC
    MrC
    Volunteer Moderator
    edited January 2017

    @mstasim ,

    As mentioned in the README, the converter can only figure out what to do with categories that are unambiguous. The Datavault CSV export does not contain sufficient data to tell the converter what a row of CSV values contains, nor the semantics (meaning) of various fields. Look at this one for a Login:

    "my login","Address","www.example.com","Username","joe@example.com","Password","secretstuff","a","b","c","d","e","f","g","h","i","j","k","l","m","n","notes go here"

    Notice there are only labels for "Address", "Username", and Password". Everything else after that is free-form meaningless data, except for the last column which is Notes.

    The converter can export to the following types:

    bankacct creditcard driverslicense email login membership note rewards software

    and the following Datavault types are completely indistinguishable in the CSV export, so get mapped to the category on the right:

    bank account, checking account     --> bankacct
    business contact, personal contact --> contact
    credit card, mastercard, visa      --> creditcard
    business, financial                --> business
    

    If you have customized any of the know field names, the converter will not be able to detect their meanings.

    Can you provide me a sanitized single row of a Login from the CSV that doesn't convert as a login?

  • mstasim
    mstasim
    Community Member

    Here's what came over in the Secure Note (sanitized) from the CSV. In the CSV it was Username in col1 and g@3$-!#%@gmail.com in col2, Password in col3, etc.

    Username: g@3$-!#%@gmail.com
    Password: g)^&@#%s!
    URL: https:\www.(^&%%#()^&%.com
    Secret Word: !@#$(, #@(!*

    Is there a way to manually format the CSV so it maps better (maybe with another converter?). I saw a note about modifying the converter file, but I'm not conversant in perl...

    Thanks,

    Stan

  • MrC
    MrC
    Volunteer Moderator
    edited January 2017

    Hi @mstasim,

    You're probably not going to like the amount of work it is going to require, but you can modify the CSV file in a spreadsheet. That would possibly be easier for you then doing it within 1Password. It will be your call, but I don't think you want to do this.

    Again, just to reiterate, the DataVault export, while nice to be able to get your data out, that data is essentially meaningless, and requires you the user to figure out what it all means.

    So I'll describe your options. First, let's look at the DataVault export format:

    "title goes here","label1","value1","label2,"value2","label3","value3","label4","value4","label5","value5","label6","value6","label7","value7","label8","valut8","label9","value9","label10","value10","notes go here"

    There are 22 columns in the export. Column 1 is the entry's Title, and column 22 contains the Notes. Columns 2 through 21 contain the ten Label / Value pairs supported by DataVault. The Labels are completely free-form - they are control by you when you create an item, by its Form Save feature, and by the Templates you create when you use DataVault. So there is likely very little column consistency row to row.

    Now, your options to convert and import:

    1. You can modify your labels in the CSV to match the datavault converter's label requirements (and/or with your help you and I can modify the converter a small amount to capture your labels based on the list you provide me) - this is probably the easiest choice.
    2. You can modify the export CSV and make the CSV consistent. This means the columns in the CSV must be the same for all the rows. This means you will have to review your rows, move the data in each row, such that, for example, "value4" is always the URL, "value2" is always the Username, etc. And you would have to split out each of the Category types into separate CSV files. This route may be a lot of work.

    So let's focus on (1). The converter works by using pattern matching to match the labels in the DataVault CSV export. The label "patterns" in the converter are simple, and are based on the stock fields that DataVault uses. But the patterns can be updated, and this is probably pretty trivial for most cases. What I need is the list of labels from you for a given category. Here's how it works.

    In the converter, there is a table of categories, the fields it supports to map to 1Password fields, and the patterns used to detect those fields. Here's an example Login category:

        login =>                    { textname => '', fields => [
            [ 'url',                0, qr/^Address$/, ],
            [ 'username',           0, qr/^Username$/, ],
            [ 'password',           0, qr/^Password$/, ],
            [ 'email',              0, qr/^email$/, ],
    

    The word "login" you see is the DataVault "login" category (which will get mapped to 1Password's Login category), the words url, username, password and email are all 1Password internal keys for Login category items, and the weird looking things like qr/^Address$/, qr/^Username$/, ... and **qr/^email$/ **are the patterns. All we need to do is update the patterns to match your list of Labels in the CSV. So if your labels are Website and URL, we'll modify the labels we need to match to capture those (Logins are a little more tricky, and I won't explain why here).

    So what I would need from you, is a list of your labels, per field, for you Logins, and other categories that are are your CSV export that are supported by the converter. And you can see the entire list by using a text editor (not Notepad), and looking at the the table named my %card_field_specs near the top of the converter.

    This may seem overwhelming, but it we take it one category at a time, customize the converter, convert and import, you can see what is working, and what isn't, and repeat. Starting with Logins, and your most frequent categories, you'll get to Good Enough very quickly.

    If you're still with me, and this makes sense, provide a list of your Login category labels (for the url, username, password, and email) and we'll go from there. Feel free to reply here, but after that, it may make most sense to take this offline so I can send you quick updates to the datavault converter for you to try. I'm at: mike ( at ) cappella ( dot ) us

  • mstasim
    mstasim
    Community Member

    I'm with you. Here are my categories

    Username
    Password
    URL
    eMail

    The other one I use frequently is Secret Word and there doesn't seem to be a 1Password default for that...

    I'm also not opposed to doing some CSV work (aligning the columns - for example) if that will get me there faster...

  • MrC
    MrC
    Volunteer Moderator
    edited January 2017

    @mstasim,

    Great, then we'll go with a bunch of rapid datavault converter updates I'll make for you. You may even get the hang of it, and will start doing it yourself!

    I'll place an update of the Converters\Datavault.pm file for you in Testing Bits, mentioned in the converter suite thread. I'll make changes, and update it. If you prefer email, start the conversation via email, and I'll just send it to you that way (email is faster, but I respect your possible desire to remain anonymous, so offer the Dropbox route).

    What would you like done with the "Secret Word? :
    1. is it a Question/Answer like thing?
    2. do you want it to be a hidden field (like 1Password's password fields)?
    3. is this label "Secret Word" absolutely unique to your Login categories (if so, I can use it to identify your Login records)?

    1Password won't do anything with "Secret Word", but at least it can be placed into the data (the converter can also add your custom fields just by using the --addfields option on the command line.

    Edit: I've already made the updates to accommodate your labels above. Here's an example import using your labels:

    Or with Secret Word being a concealed field:

  • MrC
    MrC
    Volunteer Moderator
    edited January 2017

    @mstasim ,

    Instead of awaiting your reply for now, I've dropped the customized Datavault.pm converter into Testing Bits as a single file. Replace your Converters\Datavault.pm file with it, and you can re-run the conversion. We'll go from there after I hear from you next.

  • mstasim
    mstasim
    Community Member

    Thanks! The Secret Word does not need to be Hidden. I'll have to check and see if it is unique to my logins.

    I'm fine with splitting the logins out and loading them separately. What are you using to edit the .pm files?

  • MrC
    MrC
    Volunteer Moderator
    edited February 2017

    @mstasim ,

    I use a text editor that came from Unix environments called vim. Its available for Widows too. I don't think you'll like it, because there's a very large learning curve. I've been using it for a very long time (and its parent, and grandparent).

    It's trivial to add the hidden attribute to any field - the post of the converter I left for you sets the concealed attribute.

  • mstasim
    mstasim
    Community Member

    Spent sometime working on it tonight with no luck. The new converter didn't do any better. I keeps pulling all my logins in as secure notes with everything in the note. There were 2 (out of 550) that pulled in as logins. The only thing I could figure looking at those was they had limited column names.

    One item I did notice was that my csv starts in Row1 with an account. There is nothing in my export that indicates what type of record it is...

    I think aligning the csv looks like the best answer. Also, what are you modifying the datavault.pn file in? I could try my hand at creating my own custom converter.

  • mstasim
    mstasim
    Community Member

    Just saw your 10:21 message. Hum - maybe directly editing the .pm won't work... What would I need to do to modify the CSV files to one for each category and clean them up before import?

  • MrC
    MrC
    Volunteer Moderator

    @mstasim,

    That doesn't make sense given the column names you gave me. Look at the example below, with the column names of URL, Username, Password, and Secret Word as you indicated:

    $ cat ~/Desktop/dv.csv
    "Target : Expect More. Pay Less.","URL","https://www-secure.target.com/","Username","joe@example.com","Password","supersecret","Secret Word","secret word value","label5","value5","label6","value6","label7","value7","label8","valut8","label9","value9","label10","value10","these are my notes"
    

    and here's the conversion:

    $ perl convert_to_1p4.pl datavault -v  ~/Desktop/dv.csv
    Imported 1 item
    Exported 1 login item
    Exported 1 total item
    You may now import the file /Users/cappella/Desktop/1P_import.1pif into 1Password
    

    Notice 1 Login is exported. Are you sure you replaced the Datavault.pm file? Are you sure those are the exact column names (including case)?

  • mstasim
    mstasim
    Community Member

    I just opened my csv in notepad (v. excel) to look at the structure of each line. I don't have any of the "" in your example. Is that and issue?

    Here's a row from my csv.

    Crayola.com,Username,swww.wwwww@gmail.com,Password,zxcasdfre,URL,http://www.crayola.com/,Secret Word,,Account Number,,eMail,,PIN,,,,,,,,

  • mstasim
    mstasim
    Community Member

    I opened the datavault.pm file with wordpad and it is the one you modified for my situation. It has mstasim in the text.

  • MrC
    MrC
    Volunteer Moderator
    edited February 2017

    @mstasim ,

    No, the double quotes around null values won't matter, for the most part. As I mentioned in the README, the Datavault export does not properly quote its CSV, and this can cause trouble. But don't worry about that now.

    The converter is detecting this entry as a Bank Account record, because of the "Account Number" field. This field is a key detection field for the bankacct category. The Username, Password, URL fields are not sufficient to detect the entry as a Login, because other DataVault records also export these field names! If you don't have any non-Login records with these fields (e.g. 'email', 'frequently flyer'), then we can use Username and Password as Login detectors.

    And because there is no value in the Account Number field, you're left with no real record category of importance, so it ends up looking like nothing more than a Note (the Username, Password, and URL fields are not in the Bank Account category, so they get pushed to notes).

    Does this make sense?

  • MrC
    MrC
    Volunteer Moderator
    edited February 2017

    To add, if you can open the Data Vault export in a spreadsheet, and change all of any one of the "Username" or "Password" or "URL" labels for all of your Login records to something unique, that I can quickly change a single value in the converter reliably detect these as Logins. Alternatively, if you can ensure no other record type in your export contains these field names, the same can be accomplished the same way.

    For example, if you are going to change one of the labels, change URL to LoginURL in all of the Login records. Then its trivial to detect that label.

  • mstasim
    mstasim
    Community Member

    Ok I get what's going on. I'll change URL to LoginURL for all the logins. That's no problem.

    I can do that tonight.

    I could also insert a column and populate the record with the type... that's pretty easy on my end, but might not be on your end...

    Also, how does it determine what to put in the Notes field in 1Password?

    Thanks for the work on this...

  • MrC
    MrC
    Volunteer Moderator
    edited February 2017

    @mstasim ,

    The URL to LoginURL will work fine, as would a record type column (but that will also add the type field to the record, and you don't want to add a column, you could use an existing one of the twenty-two columns). Your call.

    The converter works like this:

    1. Collect all the Field Label / Value pairs in the record.
    2. Match the Field Labels against the table of Category to Field mappings to find the target Category - the first Field Label match that is allowed to set the Category type wins.
    3. If there is no match, or there was a match but the matched Field is not allowed to set the Category, then fallback to testing if any of the Login category Field Labels are present whereby it would be considered a Login, otherwise it is a Secure Note.
    4. There is a special case later that when the record is "normalized" to an internal form, if there are no Fields with values, the record is demoted to Secure Note.

    The reason only some Fields are allowed to set the Category is because some Field Labels are present in multiple export Categories (e.g. Username, Email, etc.).

    This treatment allows the record to be placed into the most specific Category when possible, or falls back to the most relevant Category based on the Field Labels present.

  • mstasim
    mstasim
    Community Member

    I'd go with LoginURL for the key v reading one of the other columns for the type...

    Thanks

  • MrC
    MrC
    Volunteer Moderator
    edited February 2017

    @mstasim,

    Ok, If I'm understanding you correctly, you want me to LoginURL (instead of URL) as a type-setting login category label. I'll make the change now and post it to Testing Bits.

    Edit: update posted.

    perl convert_to_1p4.pl datavault -v  ~/Desktop/dvv.csv -d
    main                : Command Line: datavault -v /Users/cappella/Desktop/dvv.csv -d
    main                : Output file: /Users/cappella/Desktop/1P_import.1pif
    print_fileinfo      : Export file info
    print_fileinfo      :   size: 142
    print_fileinfo      :   kind: ASCII text
    print_fileinfo      :   mime: text/plain; charset=us-ascii
    do_import           : ROW: 1
    do_import           :   field: Username => swww.wwwww@gmail.com
    do_import           :   field: Password => supersecret
    do_import           :   field: LoginURL => http://www.crayola.com/
    do_import           :   field: Secret Word =>
    do_import           :   field: Account Number =>
    do_import           :   field: eMail =>
    do_import           :   field: PIN =>
    find_card_type      : type detected as 'login' (key='LoginURL')
    print_record        : title: Crayola.com
                          tags:
                          key(url): LoginURL = http://www.crayola.com/
                          key(password): Password = supersecret
                          key(username): Username = swww.wwwww@gmail.com
                          notes:
    Imported 1 item
    create_pif_record   : Title: Crayola.com
    create_pif_record   :   key test(username): swww.wwwww@gmail.com
    create_pif_record   :   key test(password): supersecret
    create_pif_record   :   key test(url): http://www.crayola.com/
    Exported 1 login item
    Exported 1 total item
    You may now import the file /Users/cappella/Desktop/1P_import.1pif into 1Password
    
  • mstasim
    mstasim
    Community Member

    That worked like a charm.

    Thanks for all your help.

  • MrC
    MrC
    Volunteer Moderator

    @mtasim,

    That's great news. Enjoy 1Password!

  • Greg
    Greg
    1Password Alumni

    mstasim, I am glad to hear that MrC has managed to help you! He is awesome. :)

    Please reach out to us anytime if you have more questions about 1Password. Thank you!

    ++
    Greg

This discussion has been closed.