Import from SplashID

Options
13»

Comments

  • echo
    echo
    Community Member
    Options

    I have tried all the stpes and suggstions various users have but am still not able to inport the 558 items I have in SplashID (v7.2.2 Build 780W) to 1Password v4.4.1 (441008).

    Looks like it's going to be a tedious changeover as I can only import individual items and then manually copy and paste as it gets imported as secure notes. Part of my changoever is to use this on iOS as it has TouchID functionality.

  • sjk
    sjk
    1Password Alumni
    edited September 2014
    Options

    Hi @echo,

    I'm sorry your SplashID importing hasn't been cooperating. Just to be sure, have you followed each step for exporting/importing SplashID vID data, including exporting just a few items to see if you can get those successfully imported? Also see other information in post #52 that could be helpful.

    Still failing? Here's what to do next (from post #38):

    If you're still stuck after that please create a small redacted sample file of SplashID data that fails to import and email that as an attachment to support@agilebits.com, including a link to this topic and your forum username. That sample can be created by duplicating items in SplashID, editing/removing any personal information from the duplicates, then exporting just those items. Also try importing the sample data before emailing it to confirm 1Password 4 is rejecting it.

    After determining why that smaller data sample won't import we can better suggest how to get all of your data imported. Thanks!

  • MrC
    MrC
    Volunteer Moderator
    Options

    @sjk - I'm thinking about making a SpashID converter module for my new version of converter script (which is now wallet-module based).

    I've looked briefly a the V2 and V3 vld formats, and haven't spent too much time figuring out the differences. Do you know what they are?

    @echo - how many different card types do you have, and which ones are the most important. Also, do you have any or many customizations to the fields or custom types?

  • sjk
    sjk
    1Password Alumni
    Options

    Hi @MrC,

    I'm thinking about making a SpashID converter module for my new version of converter script (which is now wallet-module based).

    Thanks for considering this!

    I don't know differences between SplashID vID formats. There shouldn't be any concern with vID 2 format when using the recommended File > Export > SplashID vID in SplashID 6 (and later) for exporting vID 3 (and later?) format. As warned here:

    Warning: In SplashID 6, the export menu options are flipped. Choosing the vID 3 menu will create a vID 2 file. Choose the vID 2 menu option to actually create the vID 3 file needed for 1Password import.

    It's unclear to me what causes data to be exported with a SplashID vID File -v4.0 header instead of SplashID vID File -v3.0, but simply changing 4.0 to 3.0 sufficiently satisfies 1Password's SplashID importer.

  • MrC
    MrC
    Volunteer Moderator
    edited September 2014
    Options

    OK, I see what the differences are. The V3 version of the file contains a few more fields, and fixes some missing labels. I haven't seen the V4 header, but I'll see what I can find.

    The VLD (I don't have a V2 labelled menu) export is presumably supposed to be the default name for the latest export version. The other one seems mislabeled. Who knows, but these are pretty silly bugs for the developer to have not caught and fixed. And for some goofy reason, the developer decided to save one cell in the export CSV, so all the data is shifted left one cell, so the text labels above do not align when imported into a spreadsheet. No wonder so many folks have trouble with getting the data correctly imported into another program.

  • echo
    echo
    Community Member
    Options

    @sjk I tried single cards and 1Password seems to be able to import that fine.

    @MrC‌ I have quite a few different card types and some have personalization. When I tried single card import, the ones that 1Password couldn't catergorize were imported as secure notes.

  • MrC
    MrC
    Volunteer Moderator
    edited October 2014
    Options

    I'll take a look at creating a converter tomorrow.

    You'd be able to specify your own custom sections in the converter to handle your custom cards. If you have only a handful of custom cards using your custom types, it probably won't be worth your time to add the custom table entry, but if you have many, it is pretty easy and I can help you define it.

    The converter will determine the card's type, and meaning of the fields, entirely by pattern matching the strings you've used for the field names (there is more than can be done here, but this would be beyond simple user customization). If you've named fields in an ambiguous fashion, then the converter won't know one wallet entry from the next. In other words, each card must have at least one field name that uniquely identifies it as belonging to a particular type, and the table of fields for that type will drive the determination of the 1P4 destination category, and how each field is placed into that 1P4 entry (including splitting some of the fields into an entirely different 1P4 entry / category).

    Let me know if this sounds useful.

  • MrC
    MrC
    Volunteer Moderator
    edited October 2014
    Options

    I have the converter done, minus a few more card type definitions.

    [ edit: ignore the comments below in this post - see subsequent threads regarding the encoding.

    I discovered that the export to VID (version 3 and version 2) botches Unicode character output. For example, a latin é (Latin small e with acute) gets output as the hexcode 8e, and this is completely wrong. It should be e9 for UTF-16 or c3 a9 for UTF-8. If you reverse the nibbles, you get e8, which is è (Latin small e with grave), so that's curious. However, other non-ASCII characters are incorrect.

    So, there will be data loss in SplashID exports, which means you'll want to check your data after importing into 1P4.

  • AgileByte
    AgileByte
    Community Member
    edited October 2014
    Options

    Just a quick note.

    The SplashID (for Mac) seems to export using the Mac Roman convention.

    I was able to turn my SplashID vID3 file into a more normal looking file as such:

    tr '\r' '\n' <input | iconv -f MACROMAN -t UTF8 >output

    That fixes both newlines (from Mac style) and character encoding to standard newlines and UTF-8 encoding.

    I then used the built-in 1Password import for my 1,800-record-long SplashID file. It is not a pretty site, but at least it's now in 1Password.

    I'd love to hear more about your better, improved importer and use that, though.

    Cheers,

    Doug

  • AgileByte
    AgileByte
    Community Member
    edited October 2014
    Options

    For some reason my command line was botched by the forum software. Probably due to Markdown decoding. :)

    Try this:

    tr '\r' '\n' < inputfile | iconv -f MACROMAN -t UTF8 > outputfile

    ... Or I could figure out that the cog icon lets me edit the previous post. That's done too now.

  • MrC
    MrC
    Volunteer Moderator
    edited October 2014
    Options

    Thanks AgileByte,

    The SplashID (for Mac) seems to export using the Mac Roman convention.

    This was the perfect hint, and it didn't even occur to me since the Mac Roman encoding was deprecated since OS X's introduction.

    Here's the completed converter in action:

  • MrC
    MrC
    Volunteer Moderator
    edited October 2014
    Options

    @sjk‌

    It's unclear to me what causes data to be exported with a SplashID vID File -v4.0 header

    This version of the vID gets created when you export SplashID vID and select Include Attachments. It is almost identical to the v3.0 version with the exception that items with attached files have their custom field label information handled differently.

    I'm looking at what it takes to support this version too.

    Edit: the converter now seamlessly supports v4.0 as well.

  • AgileByte
    AgileByte
    Community Member
    Options

    MrC, this is great news. How do you intend to make this wonderful tool available (if at all) please? I can't wait to try it. I have almost 1,900 SplashID entries that could use a proper re-import. :)

    Cheers,

    Doug

  • MrC
    MrC
    Volunteer Moderator
    Options

    I've uploaded a zip file to my _testing folder in my AgileBits Utilities folder. The file you want is convert_to_1p4.zip. Download, extract and run against your SplashID vID export file (version 3 or 4). I have not yet written documentation for this converter.

    The script runs like this:

        perl  convert_to_1p4.pl  splashid  -v   path/to/your/SplashID_vID.vid
    

    where you specify the path to your .vld file, and you've changed directories into the convert_to_1p4 folder. The commands below assume you've extracted the zip file and have the folder convert_to_1p4 on your Desktop.

    Example:

    $ cd ~/Desktop/convert_to_1p4
    $ ls
    Converters/        JSON/              Text/              UUID/              Utils/             convert_to_1p4.pl
    $ perl  convert_to_1p4.pl  splashid  -v ~/Desktop/splashid_export.vid
    Imported 20 cards
    Exported 1 email item
    Exported 1 creditcard item
    Exported 1 server item
    Exported 2 bankacct items
    Exported 4 membership items
    Exported 9 note items
    Exported 2 login items
    Exported 20 total items
    

    Note that it is the script's first argument (splashid) that tells the script what converter module to use. It will create a 1P4_import.1pif file for you on the Desktop, which you import into 1Password. I suggest you make a Backup first of your 1Password database (File > Backup). If you are importing into an existing vault, you might want to create a test vault and see how the data is added to that vault. You can move those items into the main vault when you are satisfied.

    Feel free to email me directly at the email at the top of the script, and we can work through your issues offline if necessary.

  • AgileByte
    AgileByte
    Community Member
    Options

    Thanks, Mr. C! I will try this in the coming days with high hopes.

    Cheers,

    Doug

  • sjk
    sjk
    1Password Alumni
    Options

    Hey guys,

    The SplashID (for Mac) seems to export using the Mac Roman convention.

    >

    This was the perfect hint, and it didn't even occur to me since the Mac Roman encoding was deprecated since OS X's introduction.

    I remember sometimes having to select Mac Roman format in the 1Password 3 importer to work around that issue.

    It's unclear to me what causes data to be exported with a SplashID vID File -v4.0 header

    >

    This version of the vID gets created when you export SplashID vID and select Include Attachments. It is almost identical to the v3.0 version with the exception that items with attached files have their custom field label information handled differently.

    Good find. The import guide needs a minor update to mention excluding them when exporting… if not using your converter:

    Edit: the converter now seamlessly supports v4.0 as well.

    Nice. Does your converter do anything with attachments so 1Password is able to import them?

    Thanks for your awesome effort with this!

  • MrC
    MrC
    Volunteer Moderator
    edited October 2014
    Options

    You're welcome.

    I don't do anything with the attachments, and I haven't spent any time trying to figure out how they are encoded or encrypted. A key is exported, as is the storage location within the exported bundle. Since their documents indicate that the attachments can only be re-imported into another SplashID, I suppose they are encrypted with the wallet's master key. If that's the case, I don't get the reason for that form of export (as it seems redundant since a Backup seems to be a v4.0 vID bundle anyway).

    I've updated the archive in my _testing folder, as I've now completed the transition to the new converter module system. Along the way, I completed all the card types for each converter, and added two new converters (SplashID and OS X Keychain), so we now have 9:

    Clipperz Ewallet Handysafe Keepass2 Keepassx
    Keychain Lastpass Msecure Splashid
    

    These have all been tested on OS X - I still need to retest again on Windows.

  • sjk
    sjk
    1Password Alumni
    Options

    Not worth any effort attempting to support SplashID attachments for 1Password import. And I did file a request to expand the first step of exporting SplashID viD data in the import guide to something like:

    When exporting from SplashID, make sure to use File > Export > SplashID vID, with the Export Attachments option deselected and without any password.

    And then there were nine; sweet! One that would be useful, and unlikely to ever make the list, is RoboForm.

    When I read "card" in your posts it's tempting to think of punched cards I used to submit WATFIV programs to an IBM (or was it the CDC Cyber?) mainframe with way back when. :)

    ref: DOCS-285

  • MrC
    MrC
    Volunteer Moderator
    Options

    I'll take a look at RoboForm next.

    "Card" is a term I've used for a long time, probably it initially came from eWallet, a Rolodex, or business card contact. Various password managers use different terminology, and since I'm in the middle of each, I find myself waffling about the best term. Sometimes I use "item", sometimes "card", sometimes "entry". Any advice?

  • sjk
    sjk
    1Password Alumni
    Options

    Your hero status will rise even higher if you can, at the very least, figure out a way to get a reasonable set of usable data out of the Mac version of RoboForm. :)

    The preferred 1Password terminology would be "items", e.g. the All Items category. There are numerous mentions of items in the 1Password Glossary, while "cards" is only referring to items in the Credit Card category.

  • AgileByte
    AgileByte
    Community Member
    edited October 2014
    Options

    Ahh! I have Mac OS X 10.8.5 and it has Perl 5.12.4.

    Your Convert To 1Password v1.00 fails with an error saying Perl 5.14.0 is required. Can this requirement be loosened?

    EDIT: Never mind. I apparently can download a v5.18.2 "ActivePerl" and use that.

    Thanks,

    Doug

  • AgileByte
    AgileByte
    Community Member
    Options
    Computer:/Volumes/SID1P/onepassword-utilities/convert_to_1p4$ /usr/local/ActivePerl-5.18/bin/perl ./convert_to_1p4.pl splashid -v /Volumes/SID1P/Export-20141013.sid.vid/Export-20141013.sid.vid 
    Use of uninitialized value $label in string eq at Converters/Splashid.pm line
        257, <$io> line 9 (#1)
        (W uninitialized) An undefined value was used as if it were already
        defined.  It was interpreted as a "" or a 0, but maybe it was a mistake.
        To suppress this warning assign a defined value to your variables.
    
        To help you figure out what was undefined, perl will try to tell you
        the name of the variable (if any) that was undefined.  In some cases
        it cannot do this, so it also tells you what operation you used the
        undefined value in.  Note, however, that perl optimizes your program
        anid the operation displayed in the warning may not necessarily appear
        literally in your program.  For example, "that $foo" is usually
        optimized into "that " . $foo, and the warning will refer to the
        concatenation (.) operator, even though there is no . in
        your program.
    
    Use of uninitialized value in join or string at Converters/Splashid.pm line
        386, <$io> line 9 (#1)
    Use of uninitialized value $label in string eq at Converters/Splashid.pm line
        257, <$io> line 12 (#1)
    Use of uninitialized value in join or string at Converters/Splashid.pm line
        386, <$io> line 12 (#1)
    Use of uninitialized value $label in string eq at Converters/Splashid.pm line
        257, <$io> line 22 (#1)
    Use of uninitialized value $inkey in string eq at Converters/Splashid.pm line
        359, <$io> line 22 (#1)
    

    This goes on for, literally (thanks to wc), over 3,900 lines, and ends with:

    Imported 1814 cards
    Exported 144 membership items
    Exported 19 creditcard items
    Exported 833 login items
    Exported 818 note items
    Exported 1814 total items
    

    Any ideas about the errors? I haven't imported the output into 1Password yet to see how it worked.

    Thanks,

    Doug

  • MrC
    MrC
    Volunteer Moderator
    Options

    Hold on, I'm checking on both of your issues now...

  • MrC
    MrC
    Volunteer Moderator
    edited October 2014
    Options

    I just updated the convert_to_1p4 version in the _testing folder on my AgileBits Utilities area. This fixes an issue that would prevent the converter from working on 5.12.

    As for 5.12, I can't verify that it works, as there have been important Unicode changes beyond Perl version 5.12, but you can see how well it works for you. In my test here, it appears to work. You need to go into each file (converter_to_1p4.pl, the files in Utils, and Converter/SplashID.pm) and change the use v5.14; line at the top of the files to use v5.12;.

    I see you are using ActiveState's Perl on OS X version 5.18. I've not verified it, nor 5.18. Perl 5.18 has some changes which I have not verified as non-problematic. You are in uncharted waters here. Minimally, use the 5.16 version (instead of v5.18), but again, I've not verified that Active State's Perl works well here.

    As to the warning lines above, I can't reproduce those with the converter I'm using (which is the one posted in the _testing folder). I do check for empty Labels, and I've tried reproducing a data export that might trigger this. It is possible these are a result of 5.18, or just an error situation for which I don't have test data. I'd need you to try perl 5.16 from ActiveState, and possibly 5.12 using the above instructions using the update posted in the _testing folder.

    Which version of SplashID are you using?

    Maybe you can save me a sanitized version of a single record export of what would be the 9th data record if you imported into a spreadsheet. I'm presuming this is a custom item type or a custom item label.

  • AgileByte
    AgileByte
    Community Member
    Options

    I will try your suggestions anon and let you know the results. Thanks.

  • Newell
    Newell
    Community Member
    Options

    Thanks for all the hard work, MrC. I successfully imported a SplashID DB of 642 items in, and am now in the 'cleanup' process. To my horror, 1PW doesn't allow you to edit an item's category, instead forcing you to create a whole new item of the desired category, then cut-and-paste relevant data in, and finally deleting the old. mis-categorized item. For a few individual items this is no large effort, but for app switchers, it's fairly daunting. Your helpful conversion utility defaults many of the tricky SplashID records into Secure Notes which I prefer to correct. The built-in 1PW item import screen promisingly offers to read SplashID VID files, but did nothing for me when I tried. If the feature doesn't work, shouldn't it be left off the UI? Any advice for me? I'm thinking if I massaged my SplashID DB over in that app (which allows category edits, etc) I may get a cleaner import with your converter...

  • MrC
    MrC
    Volunteer Moderator
    edited October 2014
    Options

    Hi @Newell,

    You're welcome, and I'm glad you've at least imported your records.

    As to category switching, it is on AgileBit's List of Good Intentions. The problem appears to be more difficult than users recognize, due to how behaviors and the UI are driven by categories. Categories have specific meanings in 1Password, whereas in other password managers they are essentially free-form tables of data. That said, I recognize your plight. As a user, I've wanted to switch some records in the past as well.

    The converters I've written can be customized with relative ease to accommodate most needs, and try to do "the right thing" for source categories, placing them into one or more appropriate target (1Password) categories. I'm wondering which version of converter you used, as the converter supports the following (from the --help usage output, see the supported import and export types below):

    $ perl convert_to_1p4.pl splashid --help
    
    Usage: convert_to_1p4.pl <converter> <options> <export_text_file>
    
    converters:
        clipperz ewallet handysafe keepass2 keepassx keychain lastpass msecure safeincloud
        safewallet splashid
    
    options:
        -d or --debug              # enable debug output
        -e or --exptypes <list>    # comma separated list of one or more export types from list below
        -h or --help               # output help and usage text
        -i or --imptypes <list>    # comma separated list of one or more import types from list below
        -o or --outfile <ofile>    # use file named ofile.1pif as the output file
        -v or --verbose            # output operations more verbosely
              --nowatchtower       # do not set creation date for logins to trigger Watchtower checks
    
    supported import types:
        bankacct clothes combinations creditcard email files frequentflyer identification
        insurance membership note phonenumbers prescriptions serialnum server vehicles weblogins
    supported export types:
        bankacct creditcard email login membership note server
    

    The mapping of import to export categories is table driven, and the table can be changed, per category and/or per field within a category. If you can describe which aspects you need changed, specifically, we can work together to find something that might better work to your needs.

    Converting password manager exports is a moving target, and is a one-shot. I think the AgileBits folks are prioritizing where work is done.

  • MrC
    MrC
    Volunteer Moderator
    Options

    This post is to document the new and improved version is now generally available and discussed here:

    https://discussions.agilebits.com/discussion/30286/mrcs-convert-to-1password-utility/p1

This discussion has been closed.