Folder Sync Bugs
I am using folder sync on two macs. The app stores the main vault files at:
/Users/uname/Library/Application Support/1Password 4/Data
Both macs also sync their vault to another folder (folder sync), say:
/Users/uname/vaultsync
Then I use chronosync to sync the vault files in /Users/uname/vaultsync on both macs. Unless I modify the vault on both macs since the last sync, this is supposed to work w/o any information loss, right?
Now, I was never sure whether this would work 100% reliably when I modified the vault on both macs since the last sync. In other words, can the folder sync feature understand what has changed in /Users/uname/vaultsync/myvault and apply them to /Users/uname/Library/Application Support/1Password 4/Data/myvault?
In that case chronosync (or any other sync software) would obviously register a conflict and I would need to manually pick one as the source. This would cause information loss, right? So, first please clarify how folder sync functions in these two cases.
Now the problem seems to be more serious than this. Say I have a login already in both mac's vault. And say I update the password in one of them. And I create other logins/password in that vault, too. When I sync, the new entries get copied to the other vault but the existing password is not updated. This is a simple scenario for chronosync but something gets messed up in the process. I can even see the errors from the last modification time.
Has anybody seen something like this before? I am using the latest non-beta software.
Thanks.
1Password Version: 6.1
Extension Version: Not Provided
OS Version: 10.11.4
Sync Type: folder
Comments
-
@zipped: I just came across your post. You seem to be having the same problem as I, only I'm using beta 6.2. (Please see my post from earlier today.) I've also been looking for a way to reliably sync three Macs without Dropbox or iCloud. I used to use Wi-Fi to sync both computers to my iPhone as an intermediary - I thought that it went OK for a while, but I eventually was informed that it couldn't work. So last year I began to use ChronoSync to synchronize the folder sync files two at a time. That worked for awhile, but lately has been failing. If I come up with a solution I'll let you know.
Thanks.
0 -
Yes, I saw your post, too. Honestly speaking, I am using wifi sync to my two iphones, though I never checked actually whether they have the same issue or not. I was lazy and only checking the number of items but who knows maybe the updated info like passwords in those items were not getting updated. That would render the program pretty useless.
icloud, dropbox are not an option for me. And let me tell you something. If folder sync is not fully reliable, I am pretty sure the cloud solutions won't be, either.
0 -
I've done a lot of testing - changing, deleting, or adding items on one computer and then syncing. I've found that after running ChronoSync, if the item counts on both Macs are the same, then the databases are truly identical as well.
0 -
I would be very curious to know whether you use the older Agile Keychain format or the newer OPVault when syncing.
At the heart of it, Dropbox sync is the same as Folder Sync on a Mac. 1Password reads and writes to a folder and the actual syncing is handled by Dropbox. When Dropbox finds a file has been altered in more than one location it creates conflict files and 1Password is aware of these. In these instances we read the contents of both the file and the conflicted copy and store the differences in a new custom section of the item in question and the section is titled Conflicts. To the best of my knowledge ChronoSync does not offer the ability to do anything like this and instead you have to choose one or the other. This does have the potential to result in data loss when using ChronoSync in contrast to Dropbox.
This issue is amplified if you use the OPVault container because while the Agile Keychain format specified a single file per item in your vault, the OPVault container uses bands, just like OS X does for disk images. It means multiple items are held in each band and edits to two very different items can both be to the same band. Now imagine ChronoSync is choosing one altered band over another - it could be potentially deleting all sorts of what seem to be unrelated data but still data that is stored in the same band.
This is why if you want to use Folder Sync we do strongly suggest use of a pen/flash drive that you pass between Macs. You go from a third party sync solution trying to merge different copies of the sync container to there only being one sync container, the one on the pen drive. As the pen drive is removable we get proper notifications from OS X when it is mounted or dismounted but more importantly, they're all syncing to the same container and there is no multiple differing copies to worry about.
I can understand that a pen drive may not feel like an acceptable course of action but if ChronoSync is throwing away an entire modified band in one OPVault to use the band in the other there is the very real potential for data loss.
0 -
Thanks for your reply and explanation. I'm now engaged in two parallel threads about this topic (partly my fault), so I'll leave it up to you or one of your colleagues to decide whether to merge them.
To answer your question, I had been using the Agile Keychain format, which seemed to work. However, I recently switched to the OPVault format because there was no longer a choice when creating a new folder sync file. Your points highlighting the differences (one file per database item in the older format vs. multiple items per band in the newer format) makes sense. Is there an option to go back to the Agile Keychain format or has it been deprecated?
Regarding your suggestion to use a pen-flash drive: Would I set up folder sync on each of my three computers to reference the same OPVault file on the flash drive? If so, does that mean that I would have to mount the drive whenever I use 1Password on any of the Macs? Or could I only mount the drive when I want a sync to occur? How does 1Password respond when the file it's set up to sync to isn't found?
Finally, I'll mention that I didn't adopt the ChronoSync solution on my own - it was the recommended solution on an AgileBits web site.
I would use DropBox or iCloud in which all the syncing, including conflict resolution, is handled in the background. Unfortunately, because of security restrictions at work, neither is an option for me.Thanks!
0 -
I decided to give the flash drive method a try. First, I mounted a flash drive and set up folder sync on my iMac. That created a new opvault file on the drive. Next, I set up folder sync to the same opvault file on my MacBook Air. From then on, any changes, additions, or deletions on either computer were propagated as expected. This included changes made while the flash drive wasn't mounted, as well as changes made to the same database item on both computers.
So this may be a workable solution for me, since I can choose when I sync each computer. My only concern relates to security. If someone were to gain access to the flash drive, would they be able to view all my database data? If so, I could reduce the risk by encrypting the flash drive.
Thanks!
0 -
Hi @Radoc,
Sorry for not managing to get round to replying to your first response.
As you've discovered 1Password is pretty intelligent when it comes to syncing using a pen/flash drive. Each copy of 1Password knows when it last synced and it won't attempt to do so unless the pen drive is present. That means no annoying errors that are inevitable if you try to use a network share and part of the reason we strongly recommend against doing so. Instead 1Password knows what has and hasn't synced and keeps that all until the drive is once again present. Then it updates the OPVault and also pulls down all other changes that are in it. As there is now only one copy of the OPVault the issues of a third party application trying to handle changes in multiple copies doesn't come into play.
Both the Agile Keychain and OPVault containers' contents are encrypted. These are the same formats that we use with Dropbox and to some extent iCloud too. Our security has never relied on assuming a third party service is secure, our security is all contained in the encryption that we apply equally to the encrypted SQLite data base file that resides locally or the sync container used between copies of 1Password.
As such I wouldn't personally feel the need to encrypt the drive on top of what we already do.
If you have any further questions about this approach, AKA sneakernet, please do let us know :smile:
Sorry, I just love that word (sneakernet) :tongue:
0 -
The only piece that I'm curious about is why a networked drive couldn't serve the same purpose. If the folder sync file isn't found, 1Password just waits until it's present, as you described. Why does it matter whether the file is on a flash drive or a shared drive? I can see where this would be a problem if more than one instance of 1Password was running and attempting to access the file at the same time, but that would never be true in my use case - I would just mount the share when needed.
The only reason I'm asking is that using a share would be a bit more convenient and could be done within the confines of IT security policies at work. But if that's not possible, the flash drive sneakernet solution will definitely work.
Thanks again!
0 -
Hello @Radoc,
It's a valid question and it comes down to not so much the idea of multiple copies access a single source, although that is important too, it's more to do with reliable notifications and what 1Password does when it's unlocked.
As soon as you unlock 1Password we perform a sync for any vault configured to do so. You can easily guess why, we don't want to be showing the user out of date information. While iCloud and Dropbox are both cloud services they both work in the same way, by maintaining a copy of the data on the drive. We access the copy meaning it's always present and then their service handles synchronisation of changes between the local copy and server. With a pen/flash drive it's similar except OS X gives great notifications as to when the drive is present. We get instant notifications when it's plugged in, when it's removed and no hanging about if it isn't there - the answer comes back no straight away.
Network shares are different. They can time out and OS X will wait for a minute or so before it will report that. Until then that call is left hanging. It's not just 1Password that suffers; a while back I was trying to replicate some of what our users were experiencing and ended up having to force quit Finder it got so hung up. So there can be a big delay between unlocking your vault and sync discovering it can't complete or even start for that matter. This can also be seen if you incorrectly enter your Master Password as after it has failed to unlock your local copy my understanding is it attempts to unlock the remote data. Assuming this is correct it's nearly instantaneous because for both iCloud and Dropbox the files are on the system, it's merely a file request away. With a network share though you can end up with this delay between login attempts if the share is not accessible and we're waiting on the operating system to say it's failed.
It would be great if we could so something more robust but it isn't the first time the devs have taken a look and not found anything great and I've seen the same thing with ChronoSync if you set it up to automatically sync and it can't find the network device to connect to. I believe their solution was ChronoAgent, set up their own server and have it take over this part because shares can be a pain sometimes.
So my personal thoughts are the sneakernet approach means less frustration (in theory) but if you've got a really reliable network and the various machines don't move around and always have access then it might be sufficient. If you've got a laptop that moves around though it will cause undesirable behaviour while we wait to discover that OS X can't reach the network share for whatever reason.
We do have a feature request for Wi-Fi Sync between Macs, something that isn't possible right now and I'll add your voice to this as I think this is what you'd like to see.
ref: OPM-2091
0 -
Hi again.
I understand the potential issues with network shares. However, it could work if I mounted the share immediately prior to launching 1Password, allowed the sync to complete, quit 1Password, and unmounted the share. Admittedly I'd have to be sure to follow this sequence every time, but I was willing to jump through more hoops with ChronoSync. I may give it a shot since I have good backups of my data. I also realize that I'm poking beyond the envelope. I'll let you know how it goes.
Wi-Fi sync between Macs would be great, though I get why it's not simple or high on the priority list.
Again, thanks for your help.
0 -
I was traveling quite a bit so could not follow the thread but let me chime in.
I still prefer not to use a pendrive for sync purposes. It is a hassle and at this age, it is pretty rudimentary. My main use case is I travel about 30% of the time so need to sync my two macs when I get back to home and just prior to the next departure.
I have been using chronosync for a long time (also chronoagent) for regular files, no issues so far, but with 1password, things have changed. If there is a conflict, that means data loss. But the problem I reported was beyond that. There was no conflict but somehow 1password failed to update the password in an older item.
I use the OPVault format, maybe I should switch to the one, though I dont like the idea of having to sync 300+ files.
The way I am using chronosync is that both macs use folder sync to a folder in their local disk. Then, I use chronosync to sync that folder. In theory, if there is no conflict, the chronosync will overwrite the old one and even both macs are on and using 1password, the mac with the old vault will sync to the local folder where chronosync just copied the new vault. And it has to be updated on an item level, correct? Not file level, right? In this scenario, I would expect a fully correct sync.
This is where I got a bug or something because this somehow failed for me.
0 -
@zipped: I was doing pretty much the same as you. That is, 1Password on each of my three Macs was syncing to a local file and I was using ChronoSync to periodically sync those files, two at a time. That worked but was never quite as robust as I would have liked. Basically, I wasn't confident in the process.
From what I understand based on littlebobbytables' explanations in this thread, there are two issues:
Having multiple copies of 1Password syncing to the same file isn't the same as having each sync to a local file and then syncing those files using ChronoSync.
The structure of OPVault files has made this worse because several database items may be stored in one "band."
So the solution for those of us who can't use Dropbox or iCloud is to make each copy of 1Password sync to the same file, whether it's on a flash drive or a share. Each approach has its drawbacks. With the former, if you've changed the database on both computers, you have to sync three times to ensure that changes are propagated to each. With the latter, it's important to not have more than one copy of 1Password access the file at the same time.
I'm going to try the share method, with the caveat that the type of share may be critical. I don't know how non-Mac OS formatted drives handle the sync files, which are actually packages.
Hopefully someone from AgileBits will chime in if my analysis is incorrect.
0 -
I'm sorry to hear that you're having difficulty with Folder Sync. Folder Sync can be tricky for us to troubleshoot because we do not handle the entire process.
When you enable Folder Sync, 1Password for Mac ensures that your data stays updated in whichever folder you select. After that, it is up to the sync service that you choose to move that data from the selected folder to your other computers. The challenge comes in the way that each sync service handles conflicts. They all do it differently. We know how Dropbox works, and we've built our 1Password to work with Dropbox's conflict resolution. Unfortunately, we can't support each of these unique methods, so Folder Sync is a bit of an advanced-users "at your own risk" solution.
As much of a hassle as the USB drive solution is, it might be the most reliable, because you're not relying on a third party solution to move that data, you're taking it with you to the computer that you're currently using.
0 -
Hi,
2 feature requests below regarding WiFi sync, which seems to have recently been "broken" between computers via iPhone sneakernet.
1) Make 2-way sync work again with an iPhone as the intermediate "sneakernet" device between Macs & PCs
2) Support slave-mode WiFi sync on Macs and PCs, to a "master" WLAN server (WiFi Sync server)
My environment: 2 Macs running 10.9.5 Mavericks and 1Password 4.4.3, 2 iPhones running iOS 9.3.x and auto-updating 1Password (now at 6.4.4), and a Windows 10 Pro PC onto which I will install some version of 1Password soon.
Separate topic:
Question: I have license keys for 1Password 5 and 4 for the Mac. When I eventually upgrade to OS X 10.10 or 10.11, will I be able to upgrade to 1Password 6 for Mac for free, or will there be an upgrade charge? How about from my PC 1P4 to 1P6 when it comes out?Thanks in advance!
Warm regards, and compliments on the best and safest password manager! I couldn't live without it! (319 items!) I recommend 1Password frequently, and have won you several new customers.
MarkP
Background & Details.
After starting to have items not sync'd properly between computers using WiFi sync recently (with an iPhone as go-between as I have used it successfully for 1.5 years), I have read the above thread and some other threads with interest, looking for a reliable way to again keep 2 Macs, 2 iPhones, an Android, and a Win10 PC in sync.
Like others in the forum, I do not want to use cloud sync of any type, and WiFi Sync has been perfect until recently.
Folder sync sounds troublesome from this thread (though my NAS or a shared folder on 1 Mac would be perfect if 1Password could use either of them reliably), and I'm not very interested in using a thumb drive between machines. I don't need 1 more moving part!
Request 1: Re-enable using iPhones for WiFi-Sync-based "sneakernet" go-between to sync between computers, as it used to apparently work well until about July or August. In the future, allow Androids to also be the go-between as you improve the sync sophistication of the Android 1P app.
That would not require any "extra" hardware, and a phone is harder to misplace than a thumb drive.
If each item in the vault were timestamped, sync conflicts could be resolved by keeping the newest item (and maybe with an on-screen confirmation prompt on the receiving PC/Mac). Deletes should also be propagated intelligently, definitely with an on-screen prompt.
The problem that this would solve: I recently updated the password for an account on my wife's Mac (Mac1), and deleted some obsolete items in 1Password 4.4.3. I did a WiFi sync with my iPhone, which propagated the changes perfectly to it. I then did a WiFi sync of that iPhone to my Mac (2), and it undid the changes; the Mac2 data overwrote the iPhone with the previous (obsolete) password, and restored the obsolete and deleted items, so now my iPhone matched my Mac2 perfectly, but the changes on my wife's Mac1 could not be propagated to my Mac (nor then to my Android).
This seems to leave me without a clean way to sync the 2 Macs (though maybe folder sync via a USB drive would allow correct merging???) Being still on OS X 10.9.5, I'm still on 1P4.4.3 with 1Password.agilekeychain as the only folder sync option.
How does folder sync via a USB drive determine which version to keep?
I believe this non-propagation "up" from iPhone to Mac is new : I've updated passwords on either iPhone and either Mac many times before, and using an iPhone for WiFi sneakernet has always seemed to work correctly to update the other Mac, and then the other iPhone. I've never noticed any loss of changes before August 2016.
Did you change some algorithm in the iPhone sync in July or August, around the time you changed the WiFi sync name to WLAN Sync in the iOS App? It feels like that.
This non-sync problem also sounds a lot like what @wesleykingab (https://discussions.agilebits.com/profile/wesleykingab)
described with folder sync in the thread below, where some items didn't propagate to the other Mac, though with a completely different sync mechanism than my WiFi sync:
https://discussions.agilebits.com/discussion/62794/folder-sync-not-working-with-two-macs-on-same-networkRequest 2: Allow secondary Macs or PCs to do WiFi sync with the primary Mac or PC as the WLAN server. Again maybe some prompting would be beneficial in case of hard-to-automatically-resolve conflicts - let the receiving system prompt the user with the timestamps of the conflicting items, and allow the user to decide whether to accept the change, or keep the prior value, or create a duplicate entry with a new suffix, similar to Windows File Copy (keep or replace, or keep both and rename one).
Thanks again!
MarkP
0 -
Hi @MarkP,
Thanks for taking the time to write in with your detailed experience.
To answer your question, WLAN was a name change only. The algorithm behind it has not changed.
First, a little background on WLAN (or WiFi) sync. It was designed to be a replacement for USB sync to iOS devices. While switching back and forth to use the iOS as a "sneaker-net" device works in many cases, there may be some issues which causes conflicts, and may result in what you observed (deleted items coming back, etc.), especially when changing the sync source between different machines.
FYI, when you do upgrade your Macs, the upgrade to version 6 will be free. It may work better in your situation, but we can't guarantee it, since it's a relatively complex setup that it wasn't designed for.
Ideally, we'd love to support just putting a keychain file on a NAS, but the issue is conflict resolution. NAS' usually do not handle conflict resolution well (or at all); it's usually the last client that writes to them wins. So of you have two devices that make changes offline (e.g. away from your home), if they come back at the same time, or two connected computers make changes at the same time, one of them will get overwritten and you'll lose changes (e.g. a deleted file will come back, changes to one item will be lost because there were changes on another device, etc.). Dropbox does not do that - it manages the conflicts and preserves both copies of the data, which 1Password can then merge together.
This is why we recommend a USB stick - it is quite cumbersome, but it guarantees that only one computer is syncing with it at one time.
Now, on to your requests. We can certainly consider each of those requests, though keep in mind, for the best sync experience, one device should be the source of "truth". i.e. all other devices would sync to that one, to avoid conflicts like the ones you've experienced. This makes it more difficult to do it right, especially if one has a computer located elsewhere.
Again, thanks for taking the time to describe your situation. I realize we don't have an ideal solution for you today, but we will certainly consider it when making improvements in the future.
Regards,
Kevin0 -
Thanks, @Kevin ! You're welcome, and thanks for your complete answers. I'll plan to use the USB stick until or if you are able to offer something else. I was able to upgrade to 1P5, then 1P6 on my Mac after upgrading the OS to El Capitan. All good; nice work on clean upgrades and improvements! I haven't updated my wife's Mac OS yet, and will try the iPhone sneakernet for fun once I do.
FYI, you might have your IT folks extend the domains on your SSL certificate. I tried accessing "https://forum.agilebits.com" as a (wrong) guess, and Firefox flagged a domain error on the Cert:
"forum.agilebits.com uses an invalid security certificate. The certificate is only valid for the following names: agilebits.com, www.agilebits.com Error code: SSL_ERROR_BAD_CERT_DOMAIN"
Forum.agilebits.com does redirect to https://support.1password.com, after I added the certificate exception.
Thanks again! Best, MarkP
0 -
On behalf of Kevin, you are quite welcome! :)
I'll mention the redirect to our sys admins. Thanks for taking the time to bring it up. :+1:
UPDATE: https://forum.agilebits.com now redirects without trouble. :lol:
ref: SYS-200
0 -
@ag_kevin: Several months ago, I gave up using ChronoSync and switched to keeping my Macs in sync with a USB stick and syncing my iPhone with one of the Macs. It's not very convenient, but it's worked flawlessly until now. Tonight, after I did a Wi-Fi sync between my iMac database and my iPhone, the item counts were different (343 on the iPhone, 505 on the iMac). So I wiped the data from the iPhone using the erase advanced option and tried again. I was prompted to enter the vault password as expected, but when the sync was finished the item count on the iPhone as shown under Categories was still wrong. (The item count on the sync screen was correct, however.)
To get it to work, I had to delete 1Password completely and re-install it on the iPhone. I also noticed that when I used the erase option on the iPhone, there was a notice that WLAN sync data was kept intact. What does this mean?
Thanks!
0 -
There may be something amiss with item counts and syncing with the WLAN server. Your issue sounds a little bit like the issue described here:
If that seems like the same issue to you, would you be amenable to moving the conversation to that thread?
0 -
:+1:
0
