How do I merge duplicate entries in 1Password?

1356

Comments

  • brentybrenty

    Team Member
    edited October 2017

    @djsteen: Thanks for the feedback! I really don't think we want 1Password merging information from different websites, as that could be accident prone and a potential security/privacy issue. Similarly, as mentioned previously, when combining two items, how does 1Password know what you want to keep and what you don't (title, notes, etc.)? That said, while it's a difficult problem, it is something we're interested in ourselves, so the input is greatly appreciated! :chuffed:

  • @brenty: I'd rather keep everything in the merge, then I can remove specific line items, if necessary. Some websites use multiple URLs for complicated and/or amateur reasons. For a quick — yet outdated — example, skitch.com and evernote.com would both use Evernote login credentials when skitch.com was still its own site. Other institutions might use third-party services with the same credentials as the main site and having both URLs in the same item makes 1Password a little cleaner and organised.

  • brentybrenty

    Team Member

    @djsteen: That makes perfect sense to me as well, except in a lot of cases that would totally break 1Password for anyone's use. Using your example, you've got two logins; you merge them; now you've got two different URLs (maybe passwords and a lot of other stuff too) — what happens when you try to use it? Where should 1Password take you, and what should it fill? I definitely like having multiple URLs in items, but it's only really manageable because I put them there myself. We don't want data loss, but we also don't want to render the data unusable. And when an item has two of everything, we're back to square one with the user having to prune it manually before it is usable again. Definitely food for thought though.

  • I've been a 1P user for 3-4 years and due to the lack of these kinds of features I now have roughly 3-4 "duplicates" of EVERYTHING, I don't have the time to go ONE by ONE and delete 4000 entries, even additional sorting options would greatly improve the functionality and convenience for me. I even have a few excel sheets laying around with my exported database while I attempted to remove dupes myself a few times. Thats obviously a huge security flaw for me to be leaving those exports around, so it's ultimately got me to the point of considering alternative solutions.

  • LarsLars Junior Member

    Team Member

    @ericbaranowski - sorry to hear you've had such troubles with duplicated data. 1Password doesn't often "glitch" in this way on its own; usually duplicated data is the result of two keychains being merged, or similar operations. 1Password is very conservative with your data because the push-back we get for not making it "easier" for users by making some guesses about what data should take precedence would be nothing compared to the firestorm we'd get if actions taken by processes we'd written into 1Password resulted in data loss. And we'd deserve that firestorm, too, in such a case!

    If you're finding yourself frequently having these issues of inadvertently merged vaults or duplicated entries, there is a much better way to use 1Password. You would still have to prune or merge your duplicates yourself, but only one time -- these specific sorts of problems simply don't exist in 1password.com accounts. Try one out free for 30 days, and see what you think! :)

  • rubysuerubysue Junior Member
    edited February 2018

    I just got an account and, in trying to make a single vault, ended up with 700+ exact duplicates of passwords. This from pressing one menu option Item > Share > [Vault name] > Move . There is no way an application should let a user do something so damaging and irreversible that easily!

    If you refuse to build this functionality, please create a warning: "You have multiple passwords selected. Please check to make sure these do not exist in the new vault before you move or copy them. If moving/copying creates duplicates in the target vault, you will have no way to undo the action. 1Password does not provide a way to automatically remove duplicates. As a result, you will be forced to spend hours and hours of your life deleting them one by one if you would like clean data."

  • LarsLars Junior Member

    Team Member

    @rubysue - sorry for the confusion. That certainly shouldn't be the case. When using the "Move" command, 1Password copies the selected items from the source vault into the destination vault, then trashes the items in the source vault. If you use the command Item > Share > (vault name) > Copy instead of Move, then you will get copies of the data instead because 1Password will not perform the final move-to-trash that it does with the Move command.

    I'm not sure how having more than one copy is damaging, however -- have you lost any data? If it's a question of unwanted duplicates, you can select the source vault (the one you moved data from), select the same data set you created copies of in the destination vault in the previous step, and move them to the trash. This will give you the same result as if you'd chosen the Move command. Let us know if that works for you.

  • +1 to this feature request.

    The workaround with 1password and 1password mini anchored works, but it would be nice to be able to select 2 logins and pick "merge" which gives a workflow where all non-conflicting data entries are copied and any conflicts are presented to the user on the screen to choose 1 versus the other. A lot of mobile phone apps that merge contacts do a good job at this for examples.

  • brentybrenty

    Team Member

    @bellindj: Yeah, I can imagine that being useful, not only for Logins, for for other items as well. Thanks for the feedback! :)

  • Hi @brenty, obviously this subject is long running, updates and issue, and I may have missed a thread or two. Previously, you asked a user to explain 'how' the application filled up with duplicates - an example. Following a different AgileBits Team Member's post contained in a different discussion, I 'moved' all my Primary vault logins (from iPad in this case) to my Private vault which is part of the Family vault. I took the action when I realized my logins differed between my Mac and iPad even though I was logged in as the same user. As a result, my Private vault doubled in size and my Primary vault is now empty and deleted. The good news is my devices now all work from the same Vault and have the same information. The bad news the hundreds of duplicate entries. I think it was discussed but in case, the duplicates are of three types: duplicate logins that match 100%, duplicate logins with different URL, and duplicate logins with different login details.

    I wonder whether using the 'copy' action instead of 'move' would prompt me whether to keep both or overwrite. Since I've not tested, let's say a 'copy' command indeed recognizes a duplicate using an algorithm, for example purposes only, where the 'copy' command compares default fields (basic login details, url, maybe title) and gives the user the options to delete one or the other login, merge, or insert regardless. With a bit more finesse, the merge command allows the user to select which fields to merge. I'm envisioning a Microsoft feature that I've not used for some time - export-import of contacts. In that situation, I believe a CSV file is used. Therefore, I may be discussing to possible methods for AgileBits to consider with duplicates (merge option, import/export option). I acknowledge my ideas are very high level and would need detailed discussion especially surrounding security and passwords (encrypted) being passed into an XLS or CSV back and forth during the import/export. But it's possible. And perhaps functionality can include a 'forced' backup of the all files before taking any actions to merge or export/import. I'm confident it can be done; I'm confident AgileBits can do it.

    As many other users, I've satisfactorily used 1Password for years. There's more options and competitors now. This discussion alone doesn't make consider an alternative; however, the fact that the topic and user request dates back at least four years (using forum dates) is disturbing, frustrating and disappointing. I'm not 100% sure but I believe a competitor successfully implemented export-import including the ability to import from 1Password (don't hold me to this as I read the info some time ago).

    Anyway, as indicated at the beginning, I hope the example provided shows insight how the duplicate issue arose (in my case). As best I've read and I may have missed a solution, I need to spend quiet some time manually checking, updating and/or deleting hundreds of duplicate records. Thank you. PS - sorry for any type-o that results in a lack of clarify or misunderstanding. Thank you for taking time to read this update. PSS - as I created a forum-discussion account, I noticed a pop up window that asked whether to save the login, and, there's a tab that said 'update existing.' Even a variance of that functionality could be considered to facilitate merges, maybe.

  • brentybrenty

    Team Member

    Hi @brenty, obviously this subject is long running, updates and issue, and I may have missed a thread or two.

    @10sc2co: Thanks for reaching out! No worries. Happy to help with this. I'd be lying if I said I remembered all of it myself! :lol:

    Previously, you asked a user to explain 'how' the application filled up with duplicates - an example. Following a different AgileBits Team Member's post contained in a different discussion, I 'moved' all my Primary vault logins (from iPad in this case) to my Private vault which is part of the Family vault. I took the action when I realized my logins differed between my Mac and iPad even though I was logged in as the same user. As a result, my Private vault doubled in size and my Primary vault is now empty and deleted. The good news is my devices now all work from the same Vault and have the same information. The bad news the hundreds of duplicate entries. I think it was discussed but in case, the duplicates are of three types: duplicate logins that match 100%, duplicate logins with different URL, and duplicate logins with different login details.

    Correct. That's intentional — not that you've copied/moved the same data over multiple times, but that 1Password doesn't overwrite the data, perhaps destructively, by replacing the version of an item you want to keep with a different one you don't. The good news is that, unless you've also deleted your backups, you'll have an archive of your original vault before you deleted it and/or removed everything from it (though those items should still be in the Trash).

    I wonder whether using the 'copy' action instead of 'move' would prompt me whether to keep both or overwrite.

    It will not. When an item is copied to another vault, it gets a brand new UUID to prevent overwriting data. Overwriting is only something that should happen when syncing data, not when the user explicitly copies something.

    Since I've not tested, let's say a 'copy' command indeed recognizes a duplicate using an algorithm, for example purposes only, where the 'copy' command compares default fields (basic login details, url, maybe title) and gives the user the options to delete one or the other login, merge, or insert regardless. With a bit more finesse, the merge command allows the user to select which fields to merge. I'm envisioning a Microsoft feature that I've not used for some time - export-import of contacts. In that situation, I believe a CSV file is used. Therefore, I may be discussing to possible methods for AgileBits to consider with duplicates (merge option, import/export option). I acknowledge my ideas are very high level and would need detailed discussion especially surrounding security and passwords (encrypted) being passed into an XLS or CSV back and forth during the import/export. But it's possible. And perhaps functionality can include a 'forced' backup of the all files before taking any actions to merge or export/import. I'm confident it can be done; I'm confident AgileBits can do it.

    I hear you. And that's actually a great example, for two reasons: 1) it's a FANTASTIC feature when it works, but 2) you've probably also had hassles with merging contacts. I know I have. So if we add something like this to 1Password, there's a lot to consider, and we'd want to make sure it is less trouble rather than more. For instance, should 1Password try to figure this out and warn you before you copy items? If so, that's an extra wait while it tries to analyze and compare data to present it to you...every time you try to copy. So maybe 1Password should, perhaps less helpfully, try to suggest it after copying. I think a lot of people will be annoyed by this, because they'll wonder either why they were only asked after the fact, or why 1Password is isn't them at all — indeed, they explicitly chose to copy those items there. Finally, perhaps a better compromise is to have the user invoke a "conflict resolution" for selected items manually. Then you still run into the issue of how to present it in a way that's detailed yet user friendly, not too cursory but also not too tedious. It's a tough problem.

    As many other users, I've satisfactorily used 1Password for years. There's more options and competitors now. This discussion alone doesn't make consider an alternative; however, the fact that the topic and user request dates back at least four years (using forum dates) is disturbing, frustrating and disappointing. I'm not 100% sure but I believe a competitor successfully implemented export-import including the ability to import from 1Password (don't hold me to this as I read the info some time ago).

    I don't think December 2015 has been four years ago yet, but the timing is no coincidence: when we introduced 1Password.com memberships, first with 1Password for Teams around that time, more people would copy the same data to their account multiple times, usually from different devices. It's definitely a problem in need of a solution, but a good one. After all, it's equally important for 1Password to respect the user and perform the requested action. It's your data, and 1Password shouldn't be telling you that you shouldn't have multiple items that are very similar. I have some like that intentionally, as I have templates I use and copies of items for sharing purposes. So we need to find a good way of dealing with this, and, given that exporting is insecure, I don't think that's the way to go. And, reviewing the feedback, just on this page there are people asking for very different solutions. So I'm not sure ultimately what will be best.

    Anyway, as indicated at the beginning, I hope the example provided shows insight how the duplicate issue arose (in my case). As best I've read and I may have missed a solution, I need to spend quiet some time manually checking, updating and/or deleting hundreds of duplicate records. Thank you. PS - sorry for any type-o that results in a lack of clarify or misunderstanding. Thank you for taking time to read this update. PSS - as I created a forum-discussion account, I noticed a pop up window that asked whether to save the login, and, there's a tab that said 'update existing.' Even a variance of that functionality could be considered to facilitate merges, maybe.

    Likewise, thanks for taking the time to share your own experience and feedback. If it helps, I think it may be best (based on your description) for you to start over with your original vault and then redo the migration process. But if you don' have many items or have already sorted through the extras you copied over, that may not be necessary. Let me know.

  • Thanks for the discussion. For my current situation, I followed those other instructions and my personal vault is deleted. I don't know if it's hanging around somewhere (trash, etc). I need to validate my backup exists I as I believe it does. Since I moved the vault then deleted the personal vault, I've taken no further action. So if you can guide how to best eliminate the duplicates at this time, I'd be grateful. Final thought - on my phone the data is deleted from the primary vault but the vault container remains, if that makes any difference forward. Thanks again.

  • I am fed up with the excuses. It's like you guys are such perfectionistic that you are scared to make a merge feature IN CASE it doesn't work absolutely perfectly. Seriously, there is always an element of risk when performing merge actions perhaps but believe it or not users are not babies, they can take responsibility for taking a risk. If the worst comes to it we can just backup our vault, merge, check through entries and then restore the old vault if not happy. At least then we have a sensible option. Telling us we need to do it manually is ridiculous and archaic. Get into the modern age! making using your app easier and providing solutions to the potential problems along the way is YOUR JOB!!!! perhaps if the app cost £5 people would think fair enough, i can't expect much from the developer but asking for £50(!) this app better go downstairs and make me a flippin coffee! Guys this isn't something "to consider" this is a request not only asked for by many but something expected to be there in password apps. Lastpass does it and asks for less money. I get the impression you guys thought how can we keep making money off the app and decided to make all these team features and family things. screw all that if you haven't got the basics sorted. i couldn't care less about little tweaks here and there on much less important features like teams, etc. Im' not a database expert but i can see logically that it can't be hard to tell an app to run through user name, password and URL entries and flag up results that say "ooh look, these are the same in the following entries, what do you want to do with them?". Enough with the years of excuses - you guys have seriously been taking the pi$$ (or just our money) - get off your lazy a$$es and get it implemented! you've even happily left importing passwords from keychain, etc to 1password to a user to keep developing instead of doing it yourself! Get him on the payroll (Like he more than deserves!) and these features might actually start becoming available. Until the merge feature is present you guys have got a nerve!

  • LarsLars Junior Member

    Team Member

    @steveyg777 - thanks for the feedback. Seriously, especially because it's clear you're frustrated and annoyed, I appreciate the time you took to write this all down and let us know about it; it shows you're passionate about 1Password. To be sure, "passionate" can mean either passionate advocate or ardent opponent, and it's on us to make sure we do what we can to incorporate the wishes of the user community as much as possible so that things run in a positive rather than a negative direction.

    I do need to address a couple of things, however: first, posting standards here. Usually, because we value a genuine discussion where people (not just us) are able to say what's actually on their minds, the only time we'll take the step of redacting anything a user has said in this forum is if they post personally-identifying information. But as we want to keep this a constructive and family-friendly a venue, I need to remind you to avoid the profanity and abusive language in future replies. Thanks for your cooperation.

    In terms of the substance of your post, I actually think a merge feature is a good idea; certainly it would be helpful to those in this thread and others in the past who've requested it. But I have to take issue with one specific part of your post:

    Guys this isn't something "to consider"

    It actually is, just like nearly everything else we do here. If you've read threads in the past that concern feature requests, you'll find us saying much the same things there as well. In brief, we rarely say "no," flat-out, unless we think something is truly a bad idea (especially as it affects user security); we much more-frequently can be seen saying "probably not now." That's because it would be literally impossible for us to simply drop everything and begin work on every user/feature request we receive, not least because many of them are in diametric opposition to one another - some users want a particular feature expanded, others want it removed, etc. Even for the features which we ourselves can make a case for and for which there's noticeable user support over time, there are always multiple other factors that go into our decision about when to proceed.

    You sound as if you would be very comfortable navigating a merge function and safeguarding against potential data loss, but we can't design one with only users like you in mind. We have to make sure something like a merge function is implemented in such a way that it's not just useful but provides safeguards for users who aren't as comfortable or as familiar with things as you seem to be. Because such a feature, done wrong, would make it quite easy for a less-proficient or conscientious user to do serious, potentially permanent damage to his or her own 1Password data. Long story short, I would consider this feature something we'd like to take on, but which we need to take the time to get right from the beginning.

  • brentybrenty

    Team Member

    @10sc2co: We've just introduce an experimental "Clean Up Duplicate Items" feature (available in Help > Tools) in the latest beta of 1Password for Mac version 7. I'd love for you to try that and let me know if it helps in your case. :)

  • Hi Bret, since our last exchange, here's what's up. First, I apologize for the delay (since it was April). Due to ongoing medical issues, I've been unable to focus on these types of issues. Second, unfortunately (no grimace) I don't have a MAC that is upgradable to 10.12 (10.11.6 is best) therefore I can't test the clean up (very bummed indeed). Lastly, after using the software while duplicates exists, I noticed some of the intracies involved that I didn't see or completely comprehend. Now I do; now I see there's more involved beyond what we discussed. For me, I'm stuck on the latest V6 release and will need to manually update. Thanks for the help.

  • brentybrenty

    Team Member

    @10sc2co: No need to apologize. I hope you're feeling better! I'm sorry to hear that you won't be able to update macOS, but hopefully you'll be in better shape after a bit of spring cleaning. We're here if you have any other questions. :)

  • EmmettmclaulinEmmettmclaulin Junior Member

    Two basic feature that really NEED to be added as CORE requirements.

    1. Find Duplicates
    2. Merge Duplicates

    When you use 1P over time you inevitably generate duplicate entries. It is too much work to manually go through and change each record by copying a URL then adding that URL to a different record and then deleting one of the previous records. While I am not the person coding the software - duplicate resolution is present in all kinds of database software.

    I have been using 1Password for 10+ years.

  • LarsLars Junior Member

    Team Member

    Welcome to the forum, @Emmettmclaulin! I've adjusted your username because personal email addresses should not be usernames in this public forum. Hope that's OK.

    When you use 1P over time you inevitably generate duplicate entries.

    Not really. While it's certainly possible to do so, it's far from inevitable. If you find yourself doing a lot of this and would like to learn how not to do so, can you let us know what actions you're taking that create duplicates so frequently?

    Regarding existing duplicates, try Tools > Clean up duplicate items. NOTE: this cannot be done in the All Vaults view, it needs to be done on a vault-by-vault basis.

  • EmmettmclaulinEmmettmclaulin Junior Member

    I understand how to avoid creating duplicates and updating instead of duplicating especially given multiple version of the app - multiple browsers and even multiple extensions within the same app.......AND yes it is inevitable that you end up with duplicates. I just tried the Tools > Clean up duplicate items and it found one item among 750+ entries. Not a great result as I can see dozens of entries.

    There is a long string of requests for that functionality. I am trainer of technology in the real estate sector and am a power user with a variety of tools and platforms.

    I would also like to note that the 1 Password team is pretty arrogant in a number of responses to it's users - mine included...: it's far from inevitable." Give me a break. I have paid for upgrades and subscriptions for 10-15 years and have been an raving fan / advocate. Probably better to acknowledge my concern or issue rather than deflect with your position.

  • @Emmettmclaulin sorry, but I do take exception (as a mere user) to this comment:

    AND yes it is inevitable that you end up with duplicates.

    That is simply incorrect. I have used 1Password since at least 1Password 3 and it has never created duplicate entries—nothwithstanding that I synchronise (using WLAN and, for a different vault, Dropbox). I am sure the comment @Lars made making duplicates being far from inevitable is correct...for otherwise this forum would be overrun with complaints about duplicates. It is not.

    Stephen

  • EmmettmclaulinEmmettmclaulin Junior Member

    I am not saying that the software creates duplicates - I am saying that duplicates arise out of real life scenarios - that is inevitable. In the same way that you may end up with Bill Smith, Bill &Melinda Smith and William Smith in your address book - whatever software you use.

    • I have been using it for more than a decade as well - I have also promoted its use.
  • brentybrenty

    Team Member

    @Emmettmclaulin: Ah gotcha. While I haven't personally had that problem with my 1Password data, I can definitely relate to the "address book" example. My Contacts are always a mess, and I just don't use them enough to make it worth my while to clean that up...

    Anyway, I'm sorry that you found Lars's reply offensive. I know that wasn't his intent. He's one of the least arrogant people I know, but certainly it's easy to misconstrue things in text over the internet. I apologize on his behalf. And I'll also be happy to help if you'll give me a sense of how you're running into trouble with 1Password. Perhaps there are suggestions I can offer.

    As for the duplicate cleaning feature, correct me if I'm wrong, but the items you're referring to as "duplicates" probably aren't if it isn't cleaning them up for you. We're very conservative about this since we don't want 1Password destroying data people want. So its will only be selected by this if they are truly identical.

  • Hello folks,

    Without a merge option, 1Password shows my passwords as "reused passwords" incorrectly for different sub-domains (https://site.com vs https://www.site.com etc). That incorrectly alerts me constantly about a "security problem" that doesn't exist. Can you at least ignore (or provide an option to ignore) the warnings about password reuse at sub-domain level if merge is so hard to implement?

    Thanks.

  • brentybrenty

    Team Member

    @rando: Can you clarify? If you have multiple logins saved with the same password, they should show as "reused". If you've really got two (or more) logins with the same credentials saved for the same domain, it would be best to keep only the one you need. You can save multiple URLs in a login, if that helps.

  • @brenty: Yes that's what I need the merge feature to do. I have multiple website entries with only different URLs but the same passwords. That probably happened because of the way 1Password imports entries from Chrome. When I select multiple entries and click on an option like "Merge multiple URLs into a single entry" 1Password should go ahead and convert them to a single entry as long as they all use the same password. Manually doing that would be very painful at the moment as I have hundreds of entries and prone to errors as I can mistakenly use the entry with an older password as the one to merge to.

  • brentybrenty

    Team Member

    @rando: I get that it would be easier, but we'd really rather not have 1Password try to guess what is "good" data and what isn't, and there are a lot of edge cases.

    What if the usernames are different? While that's probably not common, many sites accept a username or email address. If 1Password encounters both, what should it do? Not sure there's a wrong answer, but that's the problem.

    What about notes and URLs? The latter especially will often be different, depending on where or when you saved it. I've got a ton of logins for sites that have many login pages, and I often prefer one over the others because they're easier to work with (no popup login forms, overlays, frames, etc.)

    And there's a ton of other stuff that people do to customize items in 1Password, from titles to tags, to completely custom fields. 1Password shouldn't be throwing out data, but keeping everything from multiple items when "merging" them could have a lot of negative side effects too. Food for thought.

  • @brenty As I said, the app can do this only if there were no conflicts. It would be perfectly safe then. 99% of entries have the same username, same password, different URL. No notes or anything. They are all Chrome imports. I need a feature that specifically works for this case. This is the common case for me and I'm sure it is for many who also start using 1Password to avoid browser password managers.

  • brentybrenty

    Team Member

    Oh absolutely. I have no doubt that's true in your case, but that isn't typical based on my experience helping other 1Password users. And if and when we introduce this type of feature we'll need to take others into account regardless. Either way, it's good to have a good sense of yours. Thank you! :)

  • 1Password team, you rock! I love the product so much. I too would like to see the "merge duplicate" function implemented, but in the meantime just wanted some positive reinforcement on this feed... you do an exceptional job, and your response to forum discussion is excellent, you should be proud of the work you do. As a website developer myself and tech assistant to many clients, i understand how patience is key. You will get there, i have no doubt. Thanks again 1Password Team.

This discussion has been closed.