Google sync - how do you fix bad uploads? [#15849]

Linwood · Post by **Linwood** » Sun Jul 28, 2019 9:42 am

I managed to get 2188 to sync to google. Sort of.

I have 3292 tracks in MM5, at least that's what it says I have.

GPM, after a successful sync, said it had 3070.

I did another sync, and it said 3080 (though MM5 said it was uploading 12 songs).

I noticed some corrupted songs, e.g. that had 15 seconds or so and would not play. I took one (One Mint Jelup by Ray Charles) and deleted it from GPM, assuming it would upload again.

Sync -- didn't upload.

It would appear that there's no reconciliation process? The sync device just keeps track of what it uploaded, and there is no check to see if it is (still) on GPM?

So this begs a couple questions.

One is how, for a case like this (or for whatever reason) you delete one track, how do you make it upload again?

The other is how one reconciles to find out where the other 212 tracks are?

I'm going to try to get them in a spreadsheet but formatting and such is going to make comparison difficult. Shouldn't MM5 be doing a reconcile each time it does a sync, and upload missing?

Linwood · Post by **Linwood** » Sun Jul 28, 2019 2:35 pm

Well, there's more to this.

I found that if I expand the left panel for GPM, it has an All Tracks, which apparently is supposed to be what is uploaded.

The one track that I knew was uploaded incorrectly was not there, so I thought wow, this makes it easy to compare.

I did a COPY from it, and a COPY from the Music ALl Tracks, sorted, and did a comparison -- TONS of differences. I thought at first I had a mistake in the comparison, but I started looking and the All Tracks under GPM shows only 2780 files. There are hundreds of files that DID upload that are not present.As a simple example, the Cat Stevens Greatest Hits album was missing tracks 1, 6, and 8 in the All Tracks under MM5's list, but in GPM itself on its web page they are all present.

I found something that I thought might work, the Send-To command from the Music/All tracks (not the GPM one) had a Send To GPM option. I figured that might fill in the gaps -- nope, it immediately started showing "Copying xxx of 3292" as though it was starting over, though it is going a bit fast to be doing it (but way too slow to just be noticing it is in the list).

So basically I'm completely baffled how one is supposed to get these truly in sync.

Barry4679 · Post by **Barry4679** » Sun Jul 28, 2019 9:54 pm

Linwood wrote: ↑Sun Jul 28, 2019 9:42 am I'm going to try to get them in a spreadsheet but formatting and such is going to make comparison difficult. Shouldn't MM5 be doing a reconcile each time it does a sync, and upload missing?

No need to break out the spreadsheet just yet

What to do isn't very obvious ... Hopefully they will take your confusion on board, and see whether they can do better. Either with the UI or some training materials.

Here is what I have learnt ... But firstly a health warning. ... I let the Google Music Manager keep my GPM library in sync with my music collection ... mostly because when I first noticed that the MM5 beta went into this area, I had already allowed the Google Music Manager to transcode from FLAC to mp3, and then upload, my 44,000+ tracks to Google servers ... and I didn't want to have to do this all over again

So what I am saying is, a part of what I am saying is conjecture, because I haven't used or tried that part of this.

Sync to GPM

The aim is to:

let MM5 know what is in your GPM library

allow MM5 to store the GPM "address" of your GPM copies of your tracks

and in your case, send to GPM, any tracks that are in your collection, but which are missing from your GPM library

You use this screen (the numbers below relate to that diagram):

#1. read the dictionary of your GPM tracks ... match these to your MM5 db, and for all matches store the GPM track address into the MM db ... so that you can play from GPM using MM, or can produce GPM playlists using MM

#2. copy into GPM, any tracks not already stored at GPM

#3. upload the tracks (ie. the mp3 files) for any tracks in your GPM library, but not already stored locally in your PC or NAS ... leave this off initially, especially if you may have a lot of crap in your GPM library

#4. update your MM library, and presumably your track tags, with whatever tags are in your GPM library ... this is defaulted on, with is wrong IMO ... especially if you allow the Google Music Manager to upload tracks to your GPM library ... Google seem to minimise the data traffic cost of the upload, by finger printing your tracks, and then avoiding the upload of any tracks which they already have in their servers ... so these may not be "your" tags ... so this option will pollute your MM db tags, and probably your track tags also ... even if you don't allow the Google Music manager to upload your tracks, who knows what optimisation Google does inside their servers? ... they are storing "your" tracks at no charge ... I would guess that they consolidate tracks ... so I would leave this option off forever ... and this should be the system default (hello Ludek )

#5. Apply|Store your changes to 1<->4 above ... then run the sync

Reconcile Sync

The aims here are:

#1. did MM5 manage to match all my local tracks to the copies in my GPM library? ... MM5 uses track tags to try and obtain the match ... a lot of work has gone into making this a smart as possible, but tag differences between the two track versions can cause tracks to remain unmatched ... the consequence is that MM5 will not have the GPM "address" of your GPM track copy, so MM5 can neither play the track from GPM, nor can you include these tracks in GPM playlists that you make in GPM

#2. do you have GPM copies for all your local tracks? ... ie. have these tracks been successfully copied to GPM

Firstly #1. Use the same screen that I referenced above. ... Click on #6 .... This downloads a fresh temporary list of the contents of your GPM db ... ie. it is not showing you the contents of your MM db ... it is showing you the current contents of your GPM library ... every time you click #6, it obtains a new updated version of the contents of your GPM library.

Using this new diagram, do the following:

#1. Using the dropdown menu shown here, open what is now called the Column Filter ... in MM4 this was called the Column Browser

#2. Configure the Column filter to have these three columns ... right click each heading cell, and switch it to the columns shown in the above diagram ... the "Source" column is within the "Properties" section

#3. select this row (tool tip says "link", and the icon is meant to represent that this track is in the cloud only) in the Column Filter to show GPM tracks, in the main grid below, which have NOT successfully been matched to your GPM library

#4. select this row (tool tip says "drive", and the icon is meant to represent your disk drive) in the Column Filter to show GPM tracks, in the main grid below, which have successfully been matched to your GPM library

You can select blocks of tracks, from list #3, and cause them to be downloaded from GPM via the right click menu

Secondly #2 ... ie. any tracks which have not successfully been uploaded to GPM

get out of Devices&Services>GPM

use the Media Tree to navigate to EntireLibrary>AllTracks .... you are now looking just at your MM db

setup the Column Filter as described above

this time there are 4 filter conditions;
- #1. the track is only in MM5, no cloud copy
- #2. tracks that you used MM5 to download from GPM (so are in both locations now, and MM5 knows where all versions are)
- #3. tracks only in GPM, you have their GPM "addresses", but MM doesn't know where|if you have have a local copy of the track
- #4. tracks in both locations, and MM5 knows the address of each version, ie. they have been matched OK ... you can see, in that diagram, that MM5 has successfully matched 44,641 of my tracks for me

You can also look at any track with the Track Properties inspector ... see here how it shows which locations it has found|linked the track ... click the cloud, and it will tell you where the web copy is (GPM or Dropbox, or Google Drive or One Drive).

Other things which help:

you can include the "Source" column in any of your lists displays

or you can include the "path" column, which shows whether MM has found that you have a local copy of the track

Let me know if I have forgotten anything, or if there are any errors in my understanding

Linwood · Post by **Linwood** » Mon Jul 29, 2019 6:17 am

Wow.... I'm out of town most of today, but will review in depth when I get back.

But the idea that MM5 is altering the tags I meticulously have been applying based on what Google things should be there is frightening. That's a REALLY DUMB default if true.

Ludek · Post by **Ludek** » Mon Jul 29, 2019 8:26 am

Hi, following this.

As for the default for the '[x] Sync GPM metadata with the Library'.

I think it is safe, because:
a) the files that were uploaded by Google (or another) uploader with other tags and are umatched does not corrupt your MM library tracks at all
b) the files that were uploaded by Google (or another) uploader with other tags and are matched does not rewrite your MM library track metadata until you manually change the metadata in the GPM library (e.g. you rate it etc)
c) the files that were uploaded by MM uploader are safe as the MM metadata was used (and again only editing it in GPM library propagates the changes back to MM5 library on next sync)

Linwood · Post by **Linwood** » Mon Jul 29, 2019 3:23 pm

Just getting back to this and need to review what @Barry4679 laid out, but as to @Ludek's thoughts, here is what startled me:

If it's this:

Google seem to minimise the data traffic cost of the upload, by finger printing your tracks, and then avoiding the upload of any tracks which they already have in their servers

Plus:

#4. update your MM library, and presumably your track tags, with whatever tags are in your GPM library

Now if the first part is purely google's music manager and not MM5, fine. But my fear was that when you uploaded an MP3, if Google found it already had a copy, it did not retain your special one, but rather may just keep it's one copy and use it for everyone. Apple is reported to do that. I also have read a mixture of things about how and whether Google changes tags, art, etc. I still don't quite understand if it does.

So the fear was that you might upload a track with a set of tags, and then later Google (not me manually but some internal google process) would change those tags, and it would then get sync'd back down and change my MM5 tags.

Are you saying that simply cannot happen?

I'm a database guy at heart, and I believe you should always have "one version of the truth". The copy on Google, to me, is just a copy. not sure if that is how others use it, but that's why it was rather disconcerting to hear it sync'd both ways. Two way database sync is rarely a good thing -- it may at times be a necessary thing, but it is rarely a good thing.

But... @Barry4679 ... how do you maintain playlists if using Music Manager. Music Manager works fine for me, no issues, don't need MM5. But it doesn't appear to do playlists.

Anyway, more later after I study all the great info above.

Linwood · Post by **Linwood** » Mon Jul 29, 2019 7:01 pm

@Barry4679, thank you very much. I learned a lot about MM5 just reading this (maybe time to look for documentation generally, I've just been figuring it out as I go along). Your notes were perfect.

I did not have anything at GPM when I started, so mine were all either both, or just the disk symbol (MM5 only). I had 474 (of about 3200) files that were in MM5 and not GPM.

Which begs the question -- why not?

I checked the "Scan GPM content to the local database" and "Only include content that matches", leaving the other boxes unchecked. When I hit "sync" it runs and says it updated a few files. If I hit Scan again -- it says a few more. Again -- says 1 file. Finally said zero.

Then I sync'd again, it ran a while, said 3292, all in the cloud symbol.

So it appears to have finally caught up, but it was NOT magic. There's something that obviously would happen on each attempt, but how does one find out what?

I think over night I'll wipe it and start completely over and see what happens. in particular I'll set it to automatic and let it run over and over itself, see if it ever actually finishes.

I also need to see if this time I get bad tracks, like it did the first time, which are corrupted. Which begs the question -- does MM5 do anything to verify that it's uploads are good, e.g. on subsequent sync's, that the track has the right attributes (e.g. length)?

Anyway, thank you very much for the tutorial. Research now makes much more sense.

Barry4679 · Post by **Barry4679** » Mon Jul 29, 2019 9:11 pm

Linwood wrote: ↑Mon Jul 29, 2019 7:01 pm @Barry4679, thank you very much. I learned a lot about MM5 just reading this (maybe time to look for documentation generally, I've just been figuring it out as I go along). Your notes were perfect.

Thanks for feedback. Good luck with your search for other documentation ... this is in short supply ATM, roughly equivalent to the known supply of unicorn horns.

I did forget to add one point, although it is obvious I suppose

When|if you find that MM has not been able to (tag) match the GPM version of a track, to your local copy, you fix this by retagging either one of the copies, and then rerunning the sync

You can retag the GPM copy via their Music Manager

Linwood wrote: ↑Mon Jul 29, 2019 7:01 pm I checked the "Scan GPM content to the local database" and "Only include content that matches", leaving the other boxes unchecked. When I hit "sync" it runs and says it updated a few files. If I hit Scan again -- it says a few more. Again -- says 1 file. Finally said zero.

Then I sync'd again, it ran a while, said 3292, all in the cloud symbol.

I don't experience this ... I would say that something is silently breaking during the scan.

I am a very big fan of MM5, but I have several criticisms that I have been unable to get any buy-in from the MM Devs (so far). ... One is the lack of any available sync|download|upload logging of progress and exceptions, that is available to the user

There is a scan log that is available from the Scan Results dbox which shows at the end of the sync, but not at the end of a download as far as I know. You could try looking at that. You need to look at it before closing the Scan Results dbox, because it is not persisted. It is not very useful:

it lists tracks which were not matched, but does not list all of them ... my text file viewer shows that the row count of their Scan Log does not match the count on the Scan Results dbox

it doesn't say that that they were "skipped" because they could not be tag matched to tracks in your MM db

It has no summary at the top, ie. DateTime of run, number of tracks processed, and counts of tracks processed|Matched|newlyMatched|Updated|Imported|"skipped" ... the report is very long in a large library

and it is neither persisted nor archived

I believe that this report probably should show any errors that occur ... my run is clean ATM, so I can't check

You could retest, and check this log. ... Ludek needs to know if you are experiencing any silent errors

btw. I have also seen self-destructing popup messages that show any scan|sync errors in real time ... the popup closes after n seconds, but the message is also added to the Scan Log.

Linwood wrote: ↑Mon Jul 29, 2019 7:01 pm But... @Barry4679 ... how do you maintain playlists if using Music Manager. Music Manager works fine for me, no issues, don't need MM5. But it doesn't appear to do playlists.

I don't maintain playlists using Google Manager.

Historically I took advantage of Google's offer to house up to 50,000 of my tracks at no charge. I did this:

so that I could stream from my collection when away from home

and as a backup of last resort for my collection

I have my own MM add-on hobby app. I added a facility to my app, which allowed adoption of selected tracks from my GPM library, into my MM db ... ie tracks which I did not own, so that:

MM could start to track the play history for these tracks

so that I could use MM to create smart playlists including these tracks, where the selection criteria included duration since last time played

my app publishes these playlists into my GPM library ... the playlists included my own tracks, and the selected GPM tracks which I had imported

Then I noticed that the MM5 alpha was going to this area also, and that they were doing it better:

MM5 can play directly from the GPM source .. I had to use the GPM client app for these tracks

MM5 can download the tracks ... I did not attempt this, because I would concerned that I could be personally squashed by Google's big foot

I plan to continue by:

allowing the Google Manager to continue to sync newly ripped tracks to my GPM library

switching to use MM5 to sync my playlists to GPM, and also to import selected GPM tracks into my MM database

and to use the MM5 upload to fix any upload problem tracks that the Google Music Manager has screwed up by failing to upload for some reason

Barry4679 · Post by **Barry4679** » Mon Jul 29, 2019 11:54 pm

Hi Ludek

Ludek wrote: ↑Mon Jul 29, 2019 8:26 am As for the default for the '[x] Sync GPM metadata with the Library'.

I think it is safe, because: ...

I remain concerned about its safety and its utility.

Firstly what is the Use Case? I don't understand why anyone would want to update the metadata in their MM db with tags from their GPM library.

If they use MM to download tracks from GPM, this is expected and useful.

But not tracks where you just expecting to re-sync the existence and location of the GPM copy of your track into the MM db

No-one is going to retag their music using the Google GPM client are they? ... and then expect to import these changes into MM and their local tracks? ... the Google Music tagger is horrible to use, and it is very laggy due to Internet delays.

If people want to fix their metadata they would use your excellent Autotag facility, where they have the chance to preview incoming tags before accepting them.

So why is this optional sync action defaulted to on? ... You are defaulting on, something which will update my MM5 database ... this should be an opt in facility. ... I am genuinely surprised that you think differently

I know that I don't understand the Use Case, but I do clearly see the reputational damage and the abuse that will be coming your way if a sync corrupts customers' tags, and they find out some time afterwards.

You seem to argue that it is safe, because they should be my tags in GPM. I don't think so.

I downloaded an album from my GPM library. It is 49 minutes long, and their mp3 files totalled 113mb.

extrapolating from those two facts, I estimate that they are storing roughly 480gb of my uploaded transcoded tracks, at no charge

"No charge" means that I am not the "customer' ... I am the "product"

I think that they are using my tracks to reduce the upload cost to satisfy other customers, and that they (probably) store singleton copies of popular tracks to save storage space.
I can more or less prove this to be the case:

open "My Library" at GPM

view your library by Genre

I see "my" albums, with other people's genre tags

Just because MM5 uploads a track to MM with specific tags, this doesn't guarantee that they will remain that way. Maybe they have a periodic housekeeping and consolidation process.

You seem to be saying that it is "safe", also because you only import the metadata under controlled circumstances. Pardon me for doubting, but I am very protective of the work that I have done to groom my tags.

It would help if you specified exactly which elements of metadata you import, and under which circumstances you did this. ... I know that you have tried to explain this before, but I still don't understand. ... The info should also be displayed in the control's tool tip.

Thought: unless "Sync GPM metadata with the library" is meaning that you update the GPM library with tags FROM the MM database ... in which case this control is on the wrong tab, and is not worded well enough ... it should be on the "Library Content" tab, which controls all other movements in that direction.

Ludek · Post by **Ludek** » Tue Jul 30, 2019 6:31 am

Thought: unless "Sync GPM metadata with the library" is meaning that you update the GPM library with tags FROM the MM database ... in which case this control is on the wrong tab, and is not worded well enough ... it should be on the "Library Content" tab, which controls all other movements in that direction.

Yes, that's the case currently, i.e. "Sync GPM metadata with the library" means bi-directional sync.
i.e. if you change a rating of a track in MM library then it is propagated to the cloud (GPM library) and vice-versa.
The use case can be when you want to sync more MM5 instances synced to one GPM cloud or when you use the GPM app for android to see the updated metadata when playing the track on your Android phone (when travelling)

I can understand that specifically for GPM sync it could be desired to split this config and make two configs (for each direction).
Which doesn't make much sense for the other cloud types (OneDrive, Google Drive, Drop Box, MMS) where we explicitly use MM metadata storage databases and have full control over it.

Tracked as https://www.ventismedia.com/mantis/view.php?id=15849

EDIT:
Thinking about it further we could just:
1) change the "[x] Sync GPM metadata with the library"
into "[ ] Sync GPM metadata to the library" (disabled by default)
2) Move the checkbox above and under the '[ ] Scan GPM content to the local database' checkbox
3) Always sync the metadata in the MM > cloud direction

Is this what would work for you?

Linwood · Post by **Linwood** » Tue Jul 30, 2019 8:11 am

@Ludek, I like where you and @Barry4679 are heading on the sync question.

I want to also keep the thread going on synchronization by adding notes from yesterday's attempt at starting over.

I deleted the GPM device, deleted all music from GPM itself, and set it up to sync 3292 songs.

The program died after 977 were uploaded (per GPM), but when I restarted sync, it found only 764 of those songs, and now is syncing the remaining 2528. That's 213 of eventual overlap. That should be fine if indeed it just uploads those again, but it does seem to show that its initial sync step of determining what was already uploaded is not working properly, finding only 764 of the 977 that had been uploaded.

I'm going to let it finish and see what it eventually comes up with.

Barry4679 · Post by **Barry4679** » Tue Jul 30, 2019 8:19 am

Ludek wrote: ↑Tue Jul 30, 2019 6:31 am Thinking about it further we could just:
1) change the "[x] Sync GPM metadata with the library"
into "[ ] Sync GPM metadata to the library" (disabled by default)
2) Move the checkbox above and under the '[ ] Scan GPM content to the local database' checkbox
3) Always sync the metadata in the MM > cloud direction.

I think that is a good solution to the unique exposure in the GPM sync situation.

Thanks.

Ludek · Post by **Ludek** » Tue Jul 30, 2019 9:29 am

Linwood wrote: ↑Tue Jul 30, 2019 8:11 am The program died after 977 were uploaded (per GPM), but when I restarted sync, it found only 764 of those songs, and now is syncing the remaining 2528. That's 213 of eventual overlap. That should be fine if indeed it just uploads those again, but it does seem to show that its initial sync step of determining what was already uploaded is not working properly, finding only 764 of the 977 that had been uploaded.

Thx, I see it in the log B1A2A5CA
The server suddenly returned HTTP error 502 with msg "The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>Thatâ€™s all we know." and MM5 did not handle this error well.
Will be fixed as part of https://www.ventismedia.com/mantis/view.php?id=15850

Linwood · Post by **Linwood** » Tue Jul 30, 2019 5:46 pm

So continuing on the subject of how one reconciles all the various information, after the initial sync, GPM says I have 2819 songs (of what should be 3292).

Under the device, remote, clicking on the All Tracks calls up 2720 of which 1958 appear with just the local storage icon, and 782 appear with the cloud symbol plus local storage. To me that makes no sense -- it uploaded 2819, but then appears to have found only 2720, but of those can't tie back to its own database more than 782? Or... what does it mean?

So I hit sync again (this is #2), it said "scanning GPM" and then after several minutes it says "Library now has 3322 files" (no it doesn't, at least not if a file is a song), updated 1954 files (not sure where that number came from vs 1958 or 2720). It said "did not import 2 files (did not match import criteria)".

Hitting View Log shows two songs that say "updated" -- which is odd, it was 2 songs not imported, but one would think 1954 were updated". I would have expected the log to show all activity not just the ones not imported -- but if it is supposed to only show ones in error (e.g. not imported) I would have not expected it to say "updated".

And after accepting those dialogs the sync process message shows it is updating 572 files (that number makes some sense, being the real track count of 3292 less the remote track count 2720, though inconsistent with GPM's real count of 2819).

So I like to think I am pretty good with numbers, but the various reported numbers here just do not make much sense. Maybe they are badly labeled; maybe I just do not understand.

Maybe as it does multiple syncs this will all work itself out.

But it sure is confusing.

Linwood · Post by **Linwood** » Tue Jul 30, 2019 6:43 pm

So very quickly (or it seemed so), the sync finished, and now shows 3292 songs in GPM itself.

That's the good news. When I click on the remote All Tracks, it finds 3291, and still shows 581 with just the local storage symbol.

If I go to the Music, All Tracks it shows 582 files with just the local storage symbol.

So neither one seems to KNOW that it uploaded 3292 songs.

That's after the second sync. Back to the device, and I hit Sync Now again. This time it seemed to start part way, and showed something like 2800 of 3291 being scanned, it finished that quickly and said it "updated 569 files" and "did not import 4 files". Viewing the log only showed two files (different from last time) and those said "updated" (but see below, it was not really two).

After closing that dialog it ... did nothing. No further sync.

I hit Sync Now again, and it said it "has 3322" files (a completely wrong number), updated 1 files, and did not import 4 files. Viewing the log showed two different files HOWEVER I noticed there's a tiny scroll bar. Expanding the window showed it had 4 files skipped, and 1 updated, so the counts there match. It gives no indication WHY it skipped files, nor is there any clue why it keeps updating a few different files each time.

Sync Now again.

Same 3322 files it thinks it has, did not import 4 again, and this time same four.

The library only has 3292.

If I go back to the device Remote section it shows exactly 3292, not 3322. Now it shows only 15 files as local only so something is changing, and it shows 3277 with the cloud symbol plus local storage.

I guess the good news is that Google actually has 3292 songs.

The bad news is anyone trying to use these tools to tell what is happening has to be a lot smarter than I am.

Google sync - how do you fix bad uploads? [#15849]

Google sync - how do you fix bad uploads? [#15849]

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?

Re: Google sync - how do you fix bad uploads?