how to recover from synchronization failure

ILove1P
ILove1P
Community Member

Well the nightmare scenario has happened. It seems that at least two of my 1Password "local" vaults visible on two Macintoshes contain unique information -- different info in each vault on each computer. (These two example vaults are synchronized with Dropbox.) I expect that if I looked at other vaults I would find more issues. So I don't trust that any vault is in fact identical on all of my devices (multiple Macintoshes and one iOS device).

I first noticed the issue because the list of most-recently-modified objects (what you call items) did not match between my laptop and my home desktop. There were objects on the laptop that I needed and were not found on the desktop. Danger, Will Robinson, Danger!

I've been making screenshots of each of the two computers I've been looking at most closely to determine that there is a problem. Other than the OS being different between the two computers, the only other difference I see is that my desktop (10.11.6) has 1Password running with both "local" vaults and 1Password.com vaults while the laptop (10.12.6) copy of 1Password only has "local" vaults configured.

I'm most interested at this moment in hearing from AgileBits experts how to recover from this situation without losing data or having to inspect every single copy of each object than debugging what happened to cause this scenario. I expect that part of your response will be that I need to migrate this data into my 1Password.com account's vaults. Because Dropbox accuracy/functionality is outside of your control.

That would be fine, as long as I can import each "local" vault from each computer into the same 1Password.com vault and then be able to eliminate the true duplicates without having to look at each and every field to determine whether the records are superficially-but-not-truly-exactly-duplicates. Most things will be true duplicates, but given the number of computers and vaults and objects, ... I be scared!

Do you have database comparison tools which can flag similar objects and which fields differ between copies? And offer a reconciliation process that is at least semi-automated? I'm a Mac and UNIX/Linux systems administrator and can handle working with a tool that manipulates .1pif files if that is the ideal way to get to a known good copy of each vault.


1Password Version: 6.8.8 on both computers
Extension Version: 4.7.0
OS Version: 10.11.6 and 10.12.6
Sync Type: dropbox
Referrer: forum-search:how to recover from synchronization

Comments

  • AGAlumB
    AGAlumB
    1Password Alumni
    edited April 2018

    @ILove1P: There are two things that can help you here, and it sounds like they may both be right up your alley:

    1. Backup your data and then migrate it to your 1Password.com account — preferably the same vault, if that's feasible.
    2. Try the 1Password 7 beta. It has a new feature: Help > Tools > Clean Up Duplicate Items.

    It won't be perfect, as we go to a lot of trouble to avoid data loss due to false positives. If you've got items which are not identical because you've modified one of them on another device, you're going to have to sort that out yourself. 1Password can't know which copy has the information you want after all. But Getting the data in 1Password.com will prevent you from having a relapse, and the duplicate feature can help you with a lot of the cleanup. Let me know what you think.

  • ILove1P
    ILove1P
    Community Member

    @brenty - can you tell me how common my issue has been? Can you tell me whether I will still be able to access my standalone or 1Password.com vaults with version 6 after manipulating them with version 7beta? (2 of 3 primary computers are still happily running 10.11.6.) (And also the release notes make it seem like I'll have to rely on v6 in the future to deal with any standalone vaults that I retain. Sure I should stop having standalone vaults, but I'm conservative about my most precious data... :-) )

  • AGAlumB
    AGAlumB
    1Password Alumni

    can you tell me how common my issue has been?

    @ILove1P: You mean as far as sync issues with Dropbox? Not uncommon. That's nothing against Dropbox. They do a fantastic job at syncing files, but there can be conflicts when you change things on multiple devices, or other issues if you setup different vaults to sync with each other (which it sounds like is the case here). That's why we build 1Password.com: it's specifically designed to sync 1Password data, so it's easier to configure, and if there are sync issues we can address them.

    Can you tell me whether I will still be able to access my standalone or 1Password.com vaults with version 6 after manipulating them with version 7beta? (2 of 3 primary computers are still happily running 10.11.6.)

    Yep! Version 6 and 7 use the same data format.

    (And also the release notes make it seem like I'll have to rely on v6 in the future to deal with any standalone vaults that I retain. Sure I should stop having standalone vaults, but I'm conservative about my most precious data... :-) )

    Hmm. I'm not sure where you got the idea that standalone vaults don't work in version 7, but they most certainly do — otherwise I wouldn't have suggested using it to deal with the duplicates! Definitely backup your data first, as this is a beta, but I think you'll be in good shape — especially compared to trying to sort out the duplicates by hand.

    At the same time though, you may want to consider a 1Password.com membership, since it gives you access to all of the apps, the web interface, and does away with license management and sync configuration altogether — you simply login to your account to authorize a device and access your data. You can try it for free for 30 days to take advantage of all of its benefits. Either way though, let me know if there's anything else I can help with! :)

  • ILove1P
    ILove1P
    Community Member

    @brenty I did buy a 1Password.com membership a few months ago. I haven't yet migrated my data away from standalone vaults with the exception of a few examples. You know, life keeps me busy.

    I haven't yet looked at the find-duplicate feature. The user interface that I'm hoping to find is that of the generic file transfer client program where the contents of two directories are shown. The interface allows you to bi-directionally synchronize the contents by making it easy to resolve conflicts. Clearly doesn't scale well for cases beyond 2 or 3 directories.

    Hopefully it:

    • allows you to set the search scope: search for duplicates only within one vault, between two vaults, or across all vaults. (For extra credit, allows you to select a set of vaults containing 3 or more but fewer than all.)
    • has a button to display duplicate items fully expanded, side by side (perhaps up to 3 wide - I don't know how it should behave for cases beyond that (certainly a feature that is oriented at the desktop versions of the code.) )
    • visually flags fields which differ
    • allows you to easily choose one to keep and trash the others with two clicks (if you're offered a set of duplicates -- if not, then first the user clicks and shift or command-clicks to create a selection (on a Mac))
    • perhaps...for each field which differs, allows you to click the field in the item that you wish to keep, then offers a **MERGE **button.
    • double perhaps, has a preference setting for how tightly/loosely the amount of duplication is allowed. I say this because I have significant numbers of records which have accumulated over the years which were given titles of Unnamed. If the duplicate-finding logic only considers the item title field, then it will be of limited utility. I realize that it can be an uncomputable problem to consider the contents of too many fields.

    The release notes https://app-updates.agilebits.com/product_history/OPM7 contains this paragraph which I took to refer to standalone vaults, but perhaps is only about the format of vaults, no matter where stored:
    For users not using 1Password.com Accounts, the process of phasing out Agile Keychain starts with this beta. Agile Keychain sync files can no longer be created when setting up sync and 1Password 7 cannot sync to existing Agile Keychain files. A future beta will automatically disable sync for any vaults setup to sync to Agile Keychain. To avoid that and get a head start switch to OPVault today

  • Lars
    Lars
    1Password Alumni

    @ILove1P - You may find our duplicates feature more limited than what you were hoping for, then. It's a new feature, as brenty mentioned, so it's probably not in its final form. Traditionally, we've shied away from offering such a thing because we're not there with you when you're undertaking this kind of operation, and a set of algorithms - however well-constructed - can wind up making assumptions about a users' data that are different from what the user him/herself would assume, or would want. The last thing we want to do is make things worse for you regarding your most important data, and so our default has traditionally been that it's not 1Password's place to try to guess or assume which version of an item should "win," or even what constitutes a match.

    The current duplicates finder/remover is very conservative in its approach...but you may get some use out of it, depending on what you're trying to do. Let us know what you think!

    Not quite sure what you're referring to with regard to the paragraph from the release notes about the phasing out of Agile Keychains. If you meant: are we phasing out standalone vaults altogether? No. We have and will continue to support the OPVault format, which is the successor to Agile Keychain format (which as originally developed in the 2009 era when 1Password 3 for Mac was the current version). As time has gone on and Dropbox has changed their APIs, the differences in performance and reliability between Agile Keychain and OPVault has only increased (note: not security issues, but rather stability and usability), and it's just time to start nudging longer-term standalone users to start switching to a more modern, better-performing keychain in the form of OPVault. If you were asking about something more specific or different, can you elaborate for us? Thanks. :)

  • ILove1P
    ILove1P
    Community Member

    @Lars - I did not understand the difference between vault formats (Agile Keychains versus OPVault) and vault location (standalone vs 1Password.com). I thought OPVault was synonymous with 1Password.com. Thanks for clarifying with a great explanation.

  • ILove1P
    ILove1P
    Community Member

    @Lars I completely understand that the feature is not yet well-developed. I urge you to consider the functionality ideas I mentioned and discuss them with others. Perhaps re-read them? In my mind they would not be choosing which record/item wins or loses. They would instead illuminate the most likely discrepancies and give the user the tools to clean up their data. I believe that they would have many fewer errors than if they had to try to copy and paste the information in multiple records, flipping back and forth between records. I know that such tools for finding and rectifying duplicates would make data clean up easier for all, and even possible for the majority of folks who have less analytical outlooks.

    I have 97 items named "Unnamed", 21 which start with "Conflict copy of", 2 "New Password" and I'm sure there are a few more surprises. Cleaning up those records is obviously an unwelcome chore.

    Life would be better if I could, even, simply view two items simultaneously in the desktop client. Even if I could only edit one of them at a time.

    I just realized that even if my dirty data stems from my having used Dropbox for many years and that I won't be generating new problem cases after migrating to 1Password.com storage, that the duplicate finding and resolution tools I fantasize about will be useful for the 1Password for Families and 1Password for Businesses customers.

    Could you use code from OpenRefine (https://en.wikipedia.org/wiki/OpenRefine) or others (https://en.wikipedia.org/wiki/Data_cleansing#Tools) to simplify the task of finding duplicates? I just now discovered the "Duplicate Passwords" feature under the Security Audit section. It appears to search fields other than password.

  • Lars
    Lars
    1Password Alumni

    @ILove1P

    Could you use code from OpenRefine (https://en.wikipedia.org/wiki/OpenRefine) or others (https://en.wikipedia.org/wiki/Data_cleansing#Tools) to simplify the task of finding duplicates?

    Honestly? I don't know. In general, we can more or less do (nearly) anything. The much more difficult (and important) question is: what should we do? We're a small-ish team of just over 100 people, spread out across the globe (literally, from Tasmania to Germany, from Edinburgh, Scotland to Plano, Texas), so our resources aren't unlimited in terms of developer and designer cycles. We've thrived in this space for as long as we have (since 2006!) by paying close attention to that last question: should we pursue this or that feature/issue/etc? It's never an easy answer (well, rarely), and we don't take it lightly. I've no doubt at all that it would help you -- and potentially others as well. The question is: at what cost to other priorities that wouldn't be addressed to make room for this? How many people would benefit? Would there be drawbacks (either foreseeable or potential unknowns) for other users? You get the idea. I'll certainly pass along your ideas to our development team, who would be in a better position to answer those questions as regards the specifics of your suggestions themselves -- so you never know! Keep an eye on updates to see what we've done with the new duplicates feature, and let us know if you have any other ideas or suggestions. Thanks again! :)

  • AGAlumB
    AGAlumB
    1Password Alumni

    @ILove1P: I'd encourage you to try the duplicate feature, as I expect it will help a lot already, even if it doesn't have all the bells an whistles you're asking for. It's intentionally conservative at this time because it's a brand new feature, and the most important thing is that users do no lose data. We'll continue to refine it over time, and certainly we'll take your requests under consideration as we do so. But the best feedback we get is from real world use, so we wanted to put it out there for folks. Cheers! :)

This discussion has been closed.