How do I merge duplicate entries in 1Password?

1235»

Comments

  • edited January 3

    First off, just wow. I would also like to share that the merge feature would be incredibly helpful, especially for users who are importing password lists from multiple sources where there are bound to be duplicated.

    If not a merge feature, a feature that can at least tell you if a login is valid or not, and flag it for action (much like watchtower flags a weak or compromised password).

    As a previous 1password user, I had my database on dropbox - used it for a couple years, then for some reason transitioned to apple keychain as my devices were both MacBook Pro & iPhone which I did for a few years. But at work, I use windows and found myself unable to use my keychain, ended up using google chrome to store my passwords. Years later of having build three individual databases, I'm now finally ready to synchronize and merge everything, which I managed to do today. Now I have 1,876 items. At this point, I know hundreds of them are either invalid passwords or duplicates from my google import, 1password dropbox import, and safari password import.

    For example, I have 30 separate logins for Facebook and only 2 of them is valid (one with the username of my email, the other with my phone number). I have 98 logins for Google, which at most 10 are valid.

    I know at the end of the day I'm responsible for going through it all, but it would be nice if it was sorted/flagged in some way to make the data analysis easier for someone like me who has decided to throw all of their passwords back into 1 database from multiple ones. Maybe even something that suggests possible duplicates, or marking something an "old password" and making it a historic item to start consolidating lots of logins.

    I'm not sure I would've imported everything if I knew it was just going to make my selection "guessing game" making it harder to log in and therefore forcing me to spend hours on end to go through each individual entry to verify it's validity/usefulness.

  • brentybrenty

    Team Member

    @AmirTheMentor: The problem is, how does 1Password know what logins are "valid"? Reading your comments, I'm not even sure what you mean by that. It seems to me that having 30 or so Facebook logins and knowing that two of them are "valid", that it would be easy to flag the ones you want to keep as Favorite (to make them easy to spot briefly) and then do a search for "facebook" and Trash all but those two would be simple enough for a human, but significantly more difficult for a computer.

    Another challenge is that, with few exceptions, the things people keep calling "duplicates" are not actually duplicates. Truly duplicate data is relatively easier for a computer to identify. But a lot of this is wishing that 1Password could make some kind of value judgement about good, better, best, and then act on it without making a mistake and destroying any data the user actually wants. That's a tall order, and not something that (as far as I know) even machine learning giants like Google and Amazon have cracked yet. If you're having to go through and manually determine the "validity/usefulness" of data, it's even more problematic to do that programmatically.

  • SpaceGraySpaceGray
    edited January 3

    Another challenge is that, with few exceptions, the things people keep calling "duplicates" are not actually duplicates. Truly duplicate data is relatively easier for a computer to identify. But a lot of this is wishing that 1Password could make some kind of value judgement about good, better, best, and then act on it without making a mistake and destroying any data the user actually wants. That's a tall order, and not something that (as far as I know) even machine learning giants like Google and Amazon have cracked yet. If you're having to go through and manually determine the "validity/usefulness" of data, it's even more problematic to do that programmatically.

    I guess I don't see this as a challenge as it should be left up to the user, it doesn't need to be decided. For example, let's say you have a duplicate photo you're adding to Photos, it throws an error and says you have already added this photo, would you like to continue anyways?

    Given my example as well as @AmirTheMentor I don't think we are asking that these decisions be made but instead that the power to make them is given to us.

    I.E. Taking the 30 logins for Facebook, which is similar to my own issue, it's not that we have a hard time of knowing which login is valid (although if you don't clean it up then... well good luck lol), but instead that there's different components from each that we want to keep.

    For example for me:

    • One log in has a password only that is the correct password
    • One log in has a username
    • One log in has an email address associated with the account (that is not the username)
    • One log in has an old password that I would like to have in the Password history
    • One log in has notes about something
    • One of the above has a 2-Factor Auth Code

    That's easily 5 entries that could be completely separate. If you selected to "merge" the items, it should then prompt you to choose between any duplicated data (for example, if three of those items have a password listed and they are different, 1Password should ask the user to select the correct item and archive the rest in the history).

    Although I see this as something that would take effort to make, it's honestly such a basic feature that it's kind of unbelievable to be getting such pushback on it. I think what everyone here is saying is that this is something that shouldn't need an explanation as it's so incredibly basic to what any program with data saved does.

    @AmirTheMentor: The problem is, how does 1Password know what logins are "valid"? Reading your comments, I'm not even sure what you mean by that. It seems to me that having 30 or so Facebook logins and knowing that two of them are "valid", that it would be easy to flag the ones you want to keep as Favorite (to make them easy to spot briefly) and then do a search for "facebook" and Trash all but those two would be simple enough for a human, but significantly more difficult for a computer.

    Again, this would be something that the user should be able to do, but have a cleaner interface to do it in. I.e. Present items side by side rather than having to play with 90 different items to figure out which have data you want to keep and which don't.

    Then rather than having a computer choose, simply archive the old data with it's date originally created/last modified and it's all there in one in case a mistake is made and needed later.

  • brentybrenty

    Team Member

    There are a lot of interesting ideas here, and I think this is also a good illustration of the difficulties inherent in the subject:

    I guess I don't see this as a challenge as it should be left up to the user, it doesn't need to be decided. For example, let's say you have a duplicate photo you're adding to Photos, it throws an error and says you have already added this photo, would you like to continue anyways?

    To be clear, while I understand the point you're trying to make, you're still putting the decision on the user. I think that's as it should be, since we shouldn't be preventing the user from saving whatever they want; but it shows just one example of the contradictions in what people are saying they want: be in control of their data, but at the same time cede control to software to some vague extent in order to save them some trouble. This is why 1Password doesn't do all of this currently. It's really got to be considered carefully before we even try to execute things that affect user data.

    Given my example as well as @AmirTheMentor I don't think we are asking that these decisions be made but instead that the power to make them is given to us.

    You have that power now: both to create items you end up not wanting and to remove them. What's really being asked for here is something different than that: having software automate something you're asking to have control over. I appreciate the impulse, because who likes cleaning up after themselves? I sure don't! I'm a digital packrat of the first order. It's tedious. But at the same time I'm not using "cleaner" apps because I don't want them deleting stuff I'd rather keep -- even if maybe I'd be better off without it. ;)

    I.E. Taking the 30 logins for Facebook, which is similar to my own issue, it's not that we have a hard time of knowing which login is valid (although if you don't clean it up then... well good luck lol), but instead that there's different components from each that we want to keep. For example for me:
    One log in has a password only that is the correct password
    One log in has a username
    One log in has an email address associated with the account (that is not the username)
    One log in has an old password that I would like to have in the Password history
    One log in has notes about something
    One of the above has a 2-Factor Auth Code

    It's not going to get us to a solution in the short term, but that's very helpful as a concrete example of the problem you're having. Thank you! I haven't ever done something like that myself, and have heard from few others who have. I'd be curious why you saved all of those as separate logins in the first place. But that's sort of beside the point at the moment.

    That's easily 5 entries that could be completely separate. If you selected to "merge" the items, it should then prompt you to choose between any duplicated data (for example, if three of those items have a password listed and they are different, 1Password should ask the user to select the correct item and archive the rest in the history).
    Although I see this as something that would take effort to make, it's honestly such a basic feature that it's kind of unbelievable to be getting such pushback on it. I think what everyone here is saying is that this is something that shouldn't need an explanation as it's so incredibly basic to what any program with data saved does.

    So...regardless of how "basic" it seems to you, people would have to spend time building that, testing it thoroughly, and then we'd also need to support it on an ongoing basis. That's why there's pushback: every "basic" thing we might add that would help a few dozen people means, implicitly or explicitly, saying no to a ton of other things, often that would benefit more people. ANd how often would you use this? Once? Twice? Three times? I don't think your suggestion is a bad one. But you would still have to go through and "merge" items that way. And given that is also time and energy you'd likely rather spend on something else, you're probably going to be motivated to not create a bunch more "duplicates" since you'd have to go through this all over again for each one.

    Again, this would be something that the user should be able to do, but have a cleaner interface to do it in. I.e. Present items side by side rather than having to play with 90 different items to figure out which have data you want to keep and which don't. Then rather than having a computer choose, simply archive the old data with it's date originally created/last modified and it's all there in one in case a mistake is made and needed later.

    I think that's the crux of it. I don't disagree with you on that. It would be nice to have. But it's still work for the user, and what most of the people asking for a feature to deal with "duplicates" is something to resolve these automatically. And, in either case, we'd need to divert resources away from other things we're working on in order to build something no one would actually want to use; nobody wants to deal with cleaning up data regardless of how it's presented. Certainly one presentation may be preferable to the alternatives, and it may be that we do something in this area in the future, but man that's a tough sell on so many levels.

  • I have started running into similar duplicate management issues now that I've started using 1Password more extensively to manage app logins from my phone alongside using it from the computer.

    Right now it's difficult to match credentials matched to "Linked Apps" on your phone to existing website logins, and vice versa. So I'm starting to accumulate exactly one duplicate for each login, one for each website and one for each app. This causes trouble when I'm doing a security audit, as my "Reused Passwords" category in 1Password is filled with false positives from top to bottom.

  • LarsLars Junior Member

    Team Member

    Welcome to the forum, @bnuttall847! I'm sorry for the trouble, but I'm also not sure I understand what's causing you to accumulate all these duplicates. Can you walk me through the specific steps you're taking that are resulting in duplicates of most items? In order to get a Reused Password warning, two or more items have to have the same password. If they're for similar or identical websites, can you let me know what steps you're taking that results in this? Thanks.

  • I'd love a way to merge password items. If not that, a way to flag old duplicates as 'inactive' or 'old' so they only show up when I go looking for them. I periodically make a duplicate some how, not sure if it is updating passwords, or a change in URL, etc. I can update the appropriate item, but I'd rather not trash the old it. I don't really want to have to think about whether or not I should delete the old item - it might have saved form information that I might need in the future, etc. I pay for 1Password to not have to think about that type of stuff.

  • LarsLars Junior Member

    Team Member

    Welcome to the forum, @tasdf! The problem with any type of auto-merge is that we never want to make assumptions for the user about which version of their data is the "correct" one. That can potentially be mitigated by offering choices or simply keeping all versions, perhaps divided by sorting into different sections. But then the problem (for the user) becomes: we've essentially made NO decision that will be of any benefit, leaving the actual comparison, selection and "pruning" of data to the user. That renders the entire exercise a bit pointless. It's something there's no doubt would be a useful feature, and we'll continue to evaluate it. But for now, we don't have any dedicated feature of this nature.

    What you can do, would be to create a separate vault and give it an appropriate name, then transfer any such items you don't want to throw away yet but don't want cluttering up your "active" data. Go to Preferences > Vaults and un-check it so that items from that vault don't appear in searches of all vaults. This is what I've done on my own setup: a vault called "Archives" that is detatched from the All Vaults view. Hope that helps! :)

  • This three year old thread is just silly. Each entry in 1Password has a limited number of pieces of data. If ALL the data pieces are identical, then one of the duplicates can be deleted without threat. If all of the pieces of data are NOT identical, then they are NOT duplicates, and should be handled by user by whatever means. All I would like is for 1Password to fix the first condition for true duplicates.

  • MrCMrC Community Moderator
    edited January 31

    @fbx3,

    It is not as straightforward as you imagine. Look at these two "identical" entries:

    At first glance, you'd think they were "identical". But further inspection, shows their time stamps differ. Ok, so which one to choose?

    But lets dig deeper:

    One looks like:

    {
       "URLs" : [
          {
             "overview" : {
                "url" : "https://discussions.agilebits.com",
                "label" : "website"
             }
          }
       ],
       "sectionName" : "D",
       "uuid" : "zy5zo5f5pzd7vc7ixynxcwdev4",
       "updatedAt" : 1548892262,
       "details" : {
          "fields" : [
             {
                "designation" : "username",
                "value" : "‪‬a username",
                "name" : "username",
                "type" : "T"
             },
             {
                "designation" : "password",
                "type" : "P",
                "name" : "password",
                "value" : "‪‬some other password"
             }
          ],
          "sections" : [
             {
                "name" : "linked items",
                "title" : "Related Items"
             }
          ],
          "passwordHistory" : [
             {
                "time" : 1548892175,
                "value" : "‪‬some password"
             },
             {
                "time" : 1548892262,
                "value" : "‪‬some password other password"
             }
          ]
       },
       "overview" : {
          "url" : "https://discussions.agilebits.com",
          "ps" : 80,
          "ainfo" : "‪‬a username",
          "tags" : [
             "1PIF Import 1-25-19"
          ],
          "title" : "discussions.agilebits.com"
       },
       "categoryUUID" : "001",
       "createdAt" : 1548462423
    }
    

    and the other:

    {
       "URLs" : [
          {
             "overview" : {
                "url" : "https://discussions.agilebits.com",
                "label" : "website"
             }
          }
       ],
       "createdAt" : 1548892183,
       "details" : {
          "sections" : [
             {
                "name" : "linked items",
                "title" : "Related Items"
             }
          ],
          "fields" : [
             {
                "value" : "<U+202A><U+202C>a username",
                "designation" : "username",
                "name" : "username",
                "type" : "T"
             },
             {
                "type" : "P",
                "designation" : "password",
                "name" : "password",
                "value" : "<U+202A><U+202C>some other password"
             }
          ],
          "passwordHistory" : [
             {
                "value" : "<U+202A><U+202C>some password other password",
                "time" : 1548892257
             }
          ]
       },
       "overview" : {
          "ainfo" : "<U+202A><U+202C>a username",
          "url" : "https://discussions.agilebits.com",
          "tags" : [
             "1PIF Import 1-25-19"
          ],
          "ps" : 80,
          "title" : "discussions.agilebits.com"
       },
       "updatedAt" : 1548892257,
       "uuid" : "llkbyh63djhvfc4c2agbiaopwy",
       "sectionName" : "D",
       "categoryUUID" : "001"
    }
    

    You can see several differences, such as the password history. Not shown, since I didn't fill them in, are the variances in web form fill details.

    While on the surface they look "identical" or appear as "dups", they are not. The problem isn't in detecting a few identical fields - its in creating a usable user interface to present side-by-side fields, and have normal users figure out what to do. I think it is safe to say, with the 1Password folks being a fairly intelligent lot, the problem is non-trivial.

  • brentybrenty

    Team Member

    Indeed, we really, really don't want to be throwing out data 1Password users have saved. I know I've said it before, but I will admit to have had more superficial things in mind. Certainly there is a lot that may not be obvious at a glance too.

  • LarsLars Junior Member

    Team Member

    Thanks for doing the heavy lifting on this one, @MrC - it really is not something we want to get wrong, or allow less-technical users to get wrong by accident. As brenty has put it, people trust us to protect their data, not make decisions for them about which versions of their data are correct and which aren't.

  • Is this issue, merging duplicates, approved and in the backlog and what is the expected release date?

    I really like the software, but this is becoming an unmanageable problem with a family account. I am now paying monthly fees, so I expect this issue to be resolved.

    If I don't hear a release date, I will be looking at competitive alternatives.

    Thanks-

  • LarsLars Junior Member

    Team Member

    Welcome to the forum, @johnkountz! We don't normally pre-announce roadmaps or release dates as many factors (some beyond our control) affect them. Please stay tuned. 😀

  • Hi all...

    what are my options for essentially facilitating a 3-way-merge of two items?

    I open one item:

    it is a json blob, with a bunch of kv pairs.

    there are myriad ways to:

    • Present 'json blob a'
    • Present 'json blob b'
    • Highlight the discrepancies between a and b, via a 3 way merge.
    • Utilize human interaction to derive a single canonical superset blob.
    • any kv pair that is the same, you don't need to treat as in conflict.
    • any specific sub element of either blob a or blob b may be explicitly selected by human interaction as the canonical source for that element.
    • any specific sub element of either blob a or blob b may be moved to an alternate non-conflict subelement namespace to use
    • merge all the entries
    • display the new blob and permit the user to view, or make last minute alterations or bail out completely.
    • user approves, save the new blob as 'json blob c'
      option to autodelete the now-no-longer-canonical singleton sub elements, or present a dialog to the user per subelement and ask for guidance

    leaves the user in full control.

    provides a way to natively, and intuitively manipulate and dedup their data
    maintaining guardrails around data corruption.

    deduplication of data is a good thing, right?
    wouldn't that yield smaller databases,
    fewer items to cache and synchronize in myriad places and devices.
    fewer logs to generate
    less scaffolding data
    yes, it's a small part of a bigger picture, I totally grok that. however I think the current standpoint is a little too discouraging of empowered housekeeping.. The user should be able to do what they need to do, the app doesn't need to do it FOR them.

    Give the user the ability to perform minor surgery.
    Perhaps even leave the shotgun on the table
    - Maybe leave a big sign 'DANGER: Shooting self is a terrible idea'
    Ultimately, giving the user a way to elegantly be the masters of their own destiny would seem like a good line to try and draw RE: behavior and expectation...

    You don't have to automagically do it for me, but it would be nice if you'd let me drive...
    yes, even if it's off a cliff....

    you know...
    ...
    ...for science.

    OR, if that's not easy... can you integrate with a 3-way-merge tool to interact with the user and then distinctly return a single consolidated blob?

    that would wash your hands of the problem completely, and you still give the user a way to do the needful?

    is any of this a viable way forward?

  • LarsLars Junior Member

    Team Member

    @wolfspyre - thanks for the detailed suggestion! I particularly liked the color commentary about shooting oneself being a bad idea. :)

    Since you took the time to clearly think through and make the case for what you'd like to see, I feel even more compelled than usual to be as straight-up with you as possible. So instead of saying "I'll pass this along to the dev team," (which I will) and leaving it at that, I want to go a bit further and say I don't think this is likely to happen. Why? For a few reasons. The first is that this kind of setup is considerably more work to implement properly and maintain in a way that doesn't allow less-sophisticated users to just grab the shotgun without really understanding what it can do, and get themselves into trouble. But beyond that (or I maybe should say along with that), it's also the question of how many people would actually use such a thing (even the ones who understand it and use it properly). That number is obviously not zero, but we don't get a ton of requests for this sort of feature, so I'm guessing it's not that large a number either. And when combined with the time-and-opportunity cost of setting this up right plus maintaining it, it adds up to being less likely than a lot of other priorities we've got on our plates right now. That's made even more the case by virtue of the fact that we DID just release two entirely new versions (1Password 7 for Mac and 1Password 7 for Windows) last year, and we're now in stability/fix mode, not new feature mode. As I said, I'll run it by the dev team, but I don't expect it will get a lot of traction currently. I'd put this one in the "someday/maybe; nice to have" pile, if I had to guess. But thank you again for the suggestion, and for being a 1Password user. :)

  • mgrad92mgrad92
    edited March 6

    I'm a longtime 1Password user, and one who's eagerly awaiting an answer to this problem. I think the complexity of this problem is one of the reasons it demands a solution.

    Here's my particular issue: Right now 1Password is warning me that I'm using a duplicated password for three different logins — which I sort of am, I guess. It's the active directory password for my company's network, so it's used on a variety of intranet pages.

    My problem is that while the active directory password is the same, the rest of my login info is not. For example, one page requires us to enter our usernames as, say, "mgrad92". Another page, tho', requires us to log in as "ABC.AD\mgrad92". Yet another page requires us to log in using our company email address as the username — but all three use the same companywide authentication infrastructure, so my password is the same for all three.

    Right now, none of the options I've found seem ideal:

    1. Manually merge these items. If this feels like a complex issues to ask your devs to solve, you can understand why it feels insurmountable for me to solve. One of my entries stores the one-time password required to access my company laptop. Manually copying info like that between entries seems like inviting disaster — not to mention leaving me to figure out how to create one entry with multiple primary usernames.
    2. Delete the duplicates. They're not really duplicates, right? Since the fundamentals — username, for example — are different? Sometimes even the form labels for different username entries (in form details) are identical between two pages even though the username entries themselves are different.
    3. Just ignore the warning about duplicated passwords. Not only is it bothersome to have to scroll past this warning every day, it seems like bad practice. I want to stay vigilant about hunting down passwords that truly are duplicates.

    If there's a better solution currently available, I'd be grateful if you pointed me to it. But if there isn't, it seems like there needs to be a way to either merge database entries or, if that's not possible, allow us to link them in a way that eliminates warnings about duplicated passwords.

  • brentybrenty

    Team Member

    Indeed. That sounds like a very different problem. You don't want to get rid of or merge that data. I don't understand why they make it so complicated with regard to the username when it's the same password (and presumably the same account, but maybe not behind the scenes, even if you can't change them independently). It's a very specific use case that we'll continue to evaluate with regard to Watchtower as we iterate on the design, probably offering a way to choose to dismiss warnings in those instances. It just isn't at all related to this discussion.

  • I’ll be sure to tell them you think their system is too complicated. The company is in the throes of several recent purchases/mergers, tho’, so I think their IT folks would say my 1Password problems should be more of a priority for you right now than them.

    Meanwhile, if merging items with varying credentials isn’t related to my problem, which of those three options do you recommend?

  • brentybrenty

    Team Member

    Ignore them. As I mentioned, we will "probably [be] offering a way to choose to dismiss warnings in those instances". I can't say for certain when that will be, but I suspect it will be before they let you use a single sign in across the board (I thought that was the point of "single sign in" platforms like AD, but I am not an expert). If not, that's easier for you anyway. :)

  • Given that the whole purpose behind using a password manager is to eliminate password re-use, I would think that helping us users to take the warnings seriously would be pretty important. Eliminating false-positive warnings is a big part of this. It's why so many of us users are clamouring for the feature.

    As a developer, I understand that a full-on automated merge tool is a lot of work - and I agree there are just too many ambiguities about what a 'merge' could mean to a user.

    However, one thing that would be very straightforward, would be a feature that isn't called 'merge' but rather:
    "Copy website URL from Login A to Login B and then delete Login A".

    That way, I don't have to manually copy the URL from Login A, go to Login B, paste it, go back to Login A, delete it. It's really time-consuming, and off-putting. So either I have to ignore the false-positives (and therefore generally tune-out the warnings in general) or else spend a bunch of time that I really don't want to.

  • ⬆️ What they said.

  • brentybrenty

    Team Member

    @nealm: Please don't post the same thing in multiple places. That just makes everyone wait longer for a response, including you. I've replied to you here already a bit ago, and I don't really have anything else to add to that.

  • @brenty apologies for the duplicate request - clearly however you're not yet getting them importance of this issue, despite the repeated requests.

    I noticed in the help section of the app that there are a number of open source licenses listed. Would you be open to a pull request?

  • brentybrenty

    Team Member

    @nealm: Clearly you're not getting my point about duplicate comments. :unamused: And repeated requests from the same people aren't really beneficial to anyone, including all those getting multiple notifications for them. Please refrain from being disruptive.

    Anyway, it's an interesting idea. 1Password is not open source, but you could certainly code your own tool for merging data on GitHub or something and we'd be happy to take a look at perhaps incorporating it into a future version of 1Password. Thanks for the offer. :)

  • @brenty thanks :)

    Here's a basic implementation:
    https://github.com/nealmcc/object_merge

    There's no UI there, but there's a spot to pass in a delegate for the user to resolve the conflicts.

    If it's something you genuinely are interested in pursuing, I could refine it further. If not,. No worries.

  • brentybrenty

    Team Member

    Thanks! I've added this to our issue tracker so we can take a look when there are some spare cycles. :)

    ref: apple-3697

  • sshssh Junior Member

    FWIW, I suspect many more people than you realize have the SSO AD issue. It's inherent in Office 365 use, it seems, and I certainly have it. Part of going through the duplicate password use issues, of which I have 663(!!), is that most of them are actually using the same required password in an SSO kind of way across multiple URLs. It would be nice to be able to "merge URLs" or some related function from inside the Duplicate PWs review.

  • brentybrenty

    Team Member

    I'm sure you're right, but it doesn't come up often at all -- probably because there's a simple solution in that case: use a single Login item with all the URLs you need. It's quicker to edit an existing Login item to add one more URL than it is to create yet another new Login item with URL, username, and password (at the very least). Perhaps we'll be able to offer a way to "merge" separate items in the future, but you can see from this discussion that there is a wide range of differing use cases and ideas of how that should work, and we do need to prioritize working on things that help more people first.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file