GPM integration [#15002][#14983][#15023][#14980][#14981][#14668][#14955][#14958][#14959][#14936][#14933][#14934][#15072]

[Corneloues]

Cheers Barry,

I'll look in to that right now. It's easy for us to find/solve these problems as we're more techie, but MM5 is going to have to cater for these error messages and scenarios - even if just redirecting to a web page with help/advice links as you have posted above...

One of the downsides of any API integrations are handling all the unhappy paths...

Cheers,

Roy

[Ludek]

Hi,
MAX_PER_MACHINE_USERS_EXCEEDED means that more users uses the same device id (which is MAC address of your PC by default). I've run to this error in the past when I hardcoded our device id while implementing (and thus it was used from various users accounts). Then I changed it to MAC address of the PC (as it should be) and problem was resolved.

So it seems that you are using too many accounts from the same machine (MAC address) ?
Or your MAC address is common to more users (more GPM accounts) for some reason.

EDIT: But maybe MM5 is taking a MAC address (physical address) from a virtual network interface that could be 00-00-00-00-00-00
Could you please run command line a type command

Code: Select all

 ipconfig /all

to list all your network interfaces? What is physical address of the first interface?

EDT2: I have entered the issue as https://www.ventismedia.com/mantis/view.php?id=15201 and fixed for build 2031.
i.e. I added filtering of the pseudo-interfaces + added more debug messages to see more (in case the fix isn't going to resolve your issue).

[Barry4679]

i.e. I am just fixing my algorithm to prevent from the false matches + trying to deal with the small tag damages by GPM uploader too (re-opened https://www.ventismedia.com/mantis/view.php?id=14934 ).

In 2131 it will be able to pair also titles with additional spaces,
e.g. strings 'Vague / E La Nave Va' and 'Vague/E la Nave Va' are considered as same now.

So please re-test once 2131 is available and report back. Thanks.

I notice that 2131 fixes several issues from previous reports , but I cannot progress work towards identifing generic causes of matching failures, because 2131 has broken matching in a big way.

In the previous release I had under a 1,000 mismatches. With 2131 I now have 6,878 mismatches, over the same database. ... ie. 15% of my library.

It appears to relate to the handling of special characters.

I presume that your regex expression is getting_out_of_control, hard to understand?

It seems to be mostly OK if a track starts with parentheses .. eg. (I'm Your) Hoochie Coochie Man
But not if the parenthesis is embedded in the title .. eg. Mailman (The Lying Song)
Or not when there is any other character which is special character in a regex expression ... eg this not matched ... (Are You) The One That I've Been Waiting For?
or this ... (Don't Sit Down) [Instrumental]
The problem is compounded by matching issues involving the Album tag
ie. the track (I Can't Get No) Satisfaction is matched OK on on the album "Big Hits: High Tide And Green Grass"
but (I Can't Get No) Satisfaction is not matched on the album "Hot Rocks (1964-1971)"

It is all too depressing and difficult to try and guess what the issue is
I think that it is best if I just send you my MM5 database.
All tracks should be loaded to my GPM library, so should be matched.

Let me know if you cannot see the issue causing any specific mismatch. ... I will double check the track tags in my GPM library, and if they are OK I will send you the track, so you can try it yourself ... OK?

See PM for link to my MM5 db.

I am OK to pick up again once this regression is fixed.

[Ludek]

@Corneloues
Can you confirm whether the MAX_PER_MACHINE_USERS_EXCEEDED error is resolved for you in 2032 ?

@Barry
Thanks for sharing your MM.DB, but could you please also PM me link to debug log of the matching process (so that I can see the titles in your GPM library to match) ? I am going to revise this on Monday.

To clarify the algorithm: I don't use regex (it would be slow), I am using SQLite's FTS (full text search engine), so I have to strip special characters from the GPM track title, do FTS search and compare the found tracks with the original GPM one (ignoring spaces and special characters).
The reason why you saw only 1000 mismatches (2.5 %) previously was a bug in my algorithm causing some false matches (which is worse IMHO than unmatched tracks).
Now, I corrected it and do the second check so that all alfanum characters equals in the comparing sequence (ignoring spaces and separators while comparing), so if there is a suffix added by GPM then it is _not_ matched (to prevent from false matches).

[Barry4679]

Ludek,

OK done ... thanks for the explanation.

Is reassuring that it wasn't a regression.

I looked into the log for a couple of the mismatches

Code: Select all

	
	
00003463	124.49497986	[22340] MM5 [23324](R) MediaSync: addTrack2Library: syncID: 34d52313-771c-3509-a511-4a5c5147e4ff, title: Rollin' And Tumblin' (live)	
00003464	124.49510956	[22340] MM5 [23324](R) GetSongWithSameMetadata: Title: Rollin' And Tumblin' (live) ,Album: The Picturedrome, Holmfirth, March 2011 ,Artist: Jack Bruce ,Length: 396552	
00003465	124.49563599	[22340] MM5 [23324](R) DB open SQL: SELECT Songs.*  FROM Songs  WHERE Songs.ID IN (SELECT rowid FROM SongsText WHERE SongsText MATCH ' songtitle: Rollin And Tumblin live * artist: Jack Bruce* ') 	
00003466	124.49680328	[22340] MM5 [23324](R) DB lock took 16 ms : SELECT Songs.*  FROM Songs  WHERE Songs.ID IN (SELECT rowid FROM SongsText WHERE SongsText MATCH ' songtitle: Rollin And Tumblin live * artist: Jack Bruce* ') 	
00003467	124.49691010	[22340] MM5 [23324](R) MediaSync: addTrack2Library: new file excluded (based on config): sourceInfo: {"id":"34d52313-771c-3509-a511-4a5c5147e4ff","artwork_link":null,"sourceType":"cloud","service_id":"googleMusic","device_id":110}	


	
00042920	153.42437744	[22340] MM5 [23324](R) MediaSync: addTrack2Library: syncID: 07b7a94c-60f4-3e39-bdfe-cd08220cecdb, title: Ruben Sings!	
00042921	153.42443848	[22340] MM5 [23324](R) GetSongWithSameMetadata: Title: Ruben Sings! ,Album: Lost and Found ,Artist: Buena Vista Social Club ,Length: 50468	
00042922	153.42514038	[22340] MM5 [23324](R) DB open SQL: SELECT Songs.*  FROM Songs  WHERE Songs.ID IN (SELECT rowid FROM SongsText WHERE SongsText MATCH ' songtitle: Ruben Sings * artist: Buena Vista Social Club* ') 	
00042923	153.42619324	[22340] MM5 [23324](R) MediaSync: addTrack2Library: new file excluded (based on config): sourceInfo: {"id":"07b7a94c-60f4-3e39-bdfe-cd08220cecdb","artwork_link":"https://lh4.ggpht.com/LLRaHTyayf3-RcU3NkCvGHTdMZuWYPaB-EnEQXUb-f0ps5YcI4A2EvqKDwbhARZE0Gvxd961","sourceType":"cloud","service_id":"googleMusic","device_id":110}	



	
00096009	194.08570862	[22340] MM5 [23324](R) MediaSync: addTrack2Library: syncID: 4dd031c8-de36-3b98-bd95-55362d557244, title: Runaway Child (Minors Beware)	
00096010	194.08578491	[22340] MM5 [23324](R) GetSongWithSameMetadata: Title: Runaway Child (Minors Beware) ,Album: Conflict & Catalysis: Productions & Arrangements 1966-2006 ,Artist: Necessaries ,Length: 155740	
00096011	194.08631897	[22340] MM5 [23324](R) DB open SQL: SELECT Songs.*  FROM Songs  WHERE Songs.ID IN (SELECT rowid FROM SongsText WHERE SongsText MATCH ' songtitle: Runaway Child Minors Beware * artist: Necessaries* ') 	
00096012	194.08711243	[22340] MM5 [23324](R) MediaSync: addTrack2Library: new file excluded (based on config): sourceInfo: {"id":"4dd031c8-de36-3b98-bd95-55362d557244","artwork_link":null,"sourceType":"cloud","service_id":"googleMusic","device_id":110}

Here are the GPM and the MM5 tags for those three tracks:
https://www.dropbox.com/s/n7vj3tf724ims ... 1.png?dl=0
https://www.dropbox.com/s/d8bktuqtc2uus ... 1.png?dl=0

https://www.dropbox.com/s/bf41mfpdg0l5l ... 2.png?dl=0
https://www.dropbox.com/s/g7v41mx17n542 ... 2.png?dl=0

https://www.dropbox.com/s/p9mrebzp35g42 ... 3.png?dl=0
https://www.dropbox.com/s/cvzd17g9onjtf ... 3.png?dl=0

MM5 was sitting idle while I looked at the log file.
I went back to and attempted to navigate to EntireLibrary>AllTags
It froze and white screened ... the debug log was still running, si I sent you that log also.

[EDIT] one of the image links above pointed to incorrect file ... fixed

[Ludek]

Thanks, re-opened https://www.ventismedia.com/mantis/view.php?id=14934 and will look into it shortly (and test on your DB).
It seems to fail for titles for which the last character is a special char (I also see the reason already).

[Barry4679]

It seems to fail for titles for which the last character is a special char (I also see the reason already).

Looking forward to that fix. The pattern is obvious now that you point it out to me

There is another small issue that you could look into. The Devices&Services>GPM>AllTracks column browser still doesn't restore itself properly.

I don't know if all steps are relevant, but I did as follows:

• I was positioned at Devices&Services>GPM>AllTracks, with Column Browser displayed, with Source in col1, and a couple of other cols (AlbumArtist and Album).

• I navigated away to EntireLibrary>AllTracks

• I then closed MM5

• When i reopened MM5, I navigated back to Devices&Services>GPM>AllTracks

• After the read from GPM, the expected panel opened, with the Column Browser restored, but it had a problem.

The column browser's column 1 (Source) was not working as expected. ... the list contained just one entry ... "unmatched", ie. arrow with cloud icon. ... there was no "matched" filter item (speaker+cloud icon)

So I could not display a filtered view showing only matched tracks.

Overcome by toggling the column browser off, then back on.

[Barry4679]

re version 2134

Matching

A very good result. Only 2.3% of my library was unmatched after MM5 sync to my GPM library of 43,500+ tracks.

Most mismatches due to external causes.

Cause 1: The Google Music Uploader has given me a different track version.
examples:

• My David Bowie track name Aladdin Sane has been replaced by Aladdin Sane (2013 Remastered Version)

• GPM use different tags; Bowie's album Scary Monsters becomes Scary Monsters (And Super Creeps)

All easily fixed by altering tag with Google's GPM client. The mismatch affects a whole album usually, so in many cases one action fixes the whole album.

Cause 2: The Google Music Uploader has missed uploading some of my albums.

• their Uploader seems to have improved over time ... at one time it seems to have messed up the upload of some of my box sets

MM5 tools make this easy to fix, at either the track or the album level; ie right click menu ... GPM Sync

Cause 3: I have re-tagged the local copy of some of my albums after they were uploaded to GPM

Easily fixed by re-tagging with the GPM client, or delete from the GPM copy, and use MM5 to re-upload it to GPM


I spent about an hour playing with it, and already have reduced the mismatch to just 1.5% of my library.
It all works very well.
Uploads are fast, and reusyncs are fast.
Very pleased

I haven't check everything yet, as I still have 700+ tracks unmatched. There are some mismatches that I can't explain. See below images. Could you look at these please. I will also PM you a link to the tracks and my debug log of the latest sync.
https://www.dropbox.com/s/6nh31bxwomhcv ... 1.png?dl=0
https://www.dropbox.com/s/c80ckripdg4gr ... 2.png?dl=0
https://www.dropbox.com/s/ly0bd2x68iojc ... 3.png?dl=0


Download from GPM
Download from GPM opens a Web>Downloads node in Media Tree
but has a problem, or if it is WAD, it is useless
the node opens for a less than a second, and then closes, even though the download batch is still being processed
I download an album, the queue opens once for each track ... mostly just 1 entry, and maybe is visible only while the request is being added to the queue, but closes while the dl task is still in progress?

Trying to click on the node seems to make the GUI unstable ... ie. can use the back button to navigate elsewhere, but can no longer use the Media Tree to navigate ... and the wheel spins (forever?) ... msg is "Reading Files..."
3F8D4256: happened when I closed MM so that I could reboot it.

Also, requesting download of another album, while it is still handling the queue from the first request, seems to cancel the queue ... ie. the first album is incompletely downloaded

Upload to GPM

• not consistent with download from GPM ... opens no node in the Media tree

• seems to have a problem with paths containing special characters

https://www.dropbox.com/s/ztdiyhwiwmmn9 ... m.png?dl=0

Upload UI

• download has its own (prominent) right click menu item, which is good

• upload menu item is not in the same place ... is (hidden) within the second level of Send option in the right click menu ... this doesn't seem like a good idea

White Screen crashes

• happened twice ... I saw your (or someone's) advice somewhere, were it was said to leave the white screen sitting there long enough for MM5 to detect that it had become non-responsive .. I left it for 5 minutes .. no response ... can you reduce the timeout period?

The following error dbox did become visible when it stopped my powering off my PC ... I don't know whether it is related
https://www.dropbox.com/s/l1svqms2kygvr ... n.png?dl=0

Column Browser while in Devices&Services>GPM>AllTracks

Still has this problem: viewtopic.php?p=452300#p452300

Sometimes the summary footer is frozen ... ie. change source filter (eg. from arrow|cloud to speaker|cloud) ... the matrix changes as expected, but the summary trailer (ie. nnnnn files, nnnn B, nnnnn hours) does not change

The column Browser facility usefulness is degraded by the implementation of power scroll in Column Browser ... eg. enable column browser for AlbumArtist tag ... I have 790 entries in this list ... say I want to advance to Richie Havens ... so click on the browser's AlbumArtist column, and type "r" ... the column positions itself to "arthur Rubinstein" ... press "r" a few more times ... I get to "beth gibbons & Rustin Man" .... Richie Havens is still many many many clicks away .... try something different ... type "rich" quickly ... I get "keith RICHards" .... press Enter to see if that advances forward ... No ... try F3 ... No .... ... suggestion: enable Enter key and|or F3 to repeat previous search column & string

Devices&Services>GPM>AllTracks

Should this be a live view ... ie. enable Column Browser, with col 1 pointing to Source column ... navigate to rightarrow|could ... select something, and download it ... it disappears from rightarrow|cloud, but does not become visible in downarrow|cloud filter

Questions

Do you offer pause and restart of queues for requests of upload|download to GPM?
Can we view the queues?
Can we restart the interrupted queue after rebooting PC, or stopping and restarting MM5?
Could you post a link showing what it should look like?

I still think that it would good if there was a persistent log of sync and download requests. Also including any errors or failures (eg. could not transcode this track, or could create this file name, etc). Maybe a persistent file for each date that MM5 was used for syncing or downloading from GPM: eg. "2018.11.23: GPM Sync.log"

It would help us to keep our libraries managed and tidy.
It would be something we could send you if we have problems.

[Corneloues]

@Corneloues
Can you confirm whether the MAX_PER_MACHINE_USERS_EXCEEDED error is resolved for you in 2032 ?

@Ludek Sorry for the delay. Problem is indeed resolved - Thank you!

Just need to get around how to sync now. All the stuff I have in GPM is from my MM library but manually uploaded and using the Chrome extension to monitor folders (which really doesn't work at all well).

@Barry Any tips advice on how best to get things working? Got 5k files in GPM and would rather not have to start over...

Cheers,

Roy

[Corneloues]

@Ludek - appreciated the MM4 import in the latest build. Hadn't seen that before. Also wasn't sure if this only happened on first run. As MM5 is beta I still use MM4 99% of the time so it would be great to be able to do a regular import of the MM4 database.

Cheers,

Roy

[Barry4679]

@Barry Any tips advice on how best to get things working? Got 5k files in GPM and would rather not have to start over...

Sure ... the following assumes that you have uploaded all of your 5k tracks up into your GPM library ... let me know if you need more details, or this basic assumption is incorrect:

1. Delete directory C:\MediaMonkey 5 . or wherever you installed MM5
   • Install latest MM5
   • Accept its offer to import your MM4 database
   • Fix default MM5 options: Tools|Options|Library|MediaTree ... check EntireLibrary to enable that node, and check the pencil to alter it, and check the AlbumArtists node to enable that subnode also
   • Go to MediaTree node Devices&Services ... open Storage&Services and add Google Play Music
   • Configure the RemoteContent tab as shown below: #1 to cause MM5 to import the Google track keys, and tag match them against your MM tracks, and store each matched GPM key into your MM db, against its associated MM track .. #2 stops MM5 from bulk inserting new rows into your MM db for anything that it could not match (ie. either it is a GPM only track, or MM5 tag matching logic failed to find the matching track) ... #3 stops MM5 from bulk downloading to your PC, all the mp3 files for the unmatched tracks ... #4 stops MM5 from overwriting your MM tag values with whatever values are in your GPM library
   • see here
   • With these settings a MM5|GPM "sync" will just copy the google track addresses into your MM database ... so that you may play those tracks from your GPM location using MM ... or can design MM playlists, to be sync'd to your GPM library, and be played by the GPM client player while you are commuting or away from home
   • Press Apply, and then Sync Now
   When it is done:
   1. Open the EntireLibrary>AllTracks node from the Media Tree ... go to List view
   • Enable the Column Browser ... associate column 1 of the Column Browser with the "Source" database column ... it is in the Properties group
   • This source column will allow you to filter the display to show either tracks that were successfully match (speaker&cloud icon) .. or tracks that were not matched (cloud only)

[Ludek]

@Barry: FYI re the download queue issues, to be fixed as items 10 and 11 here:
https://www.ventismedia.com/mantis/view ... 133#c51622

Re upload queue: You can use the following: Select files > Send to > 'Playlist upload to GPM', add playlist 'Playlist upload to GPM' to the sync-list (Devices & Services > GPM > Library content) and click [Sync now]. Then you can resume it anytime using manual [Sync now] or using scheduled sync on the first tab of GPM config.

[Corneloues]

That's what I've been doing. Creating upload playlists and then using those to sync to GPM, just like device sync.

I have multiple playlists under a 'GPM' playlist. Mostly auto-playlists, but I have one which I add odds and sods to that don't fit in to an auto-playlist.

[Barry4679]

Re upload queue: You can use the following: Select files > Send to > 'Playlist upload to GPM', add playlist 'Playlist upload to GPM' to the sync-list (Devices & Services > GPM > Library content) and click [Sync now]. Then you can resume it anytime using manual [Sync now] or using scheduled sync on the first tab of GPM config.

What does this mean for uploads requested via right click SendTo|GPM(sync)? ... no pause and restart for these? ... no visibility of whether the request is completed? ... does a further request suspend and cancel off the still running previous request (as seems to happen with download requests)?

If so, that doesn't seem good enough ... could you change what happens inside MM5 so that these requests are handled as you describe above ... ie. you system generate a hidden playlist, and queue the request so that it protected from these problems?

And what happens for download batches? ... do these have pause and resume facility? Or do we have to monitor progress, and only power down, or terminate MM5, when the batch is complete?... can multiple requests be queued without wiping out the queue, as seems to happen at the moment? Is this a bug?

[Ludek]

What does this mean for uploads requested via right click SendTo|GPM(sync)? ... no pause and restart for these? ... no visibility of whether the request is completed? ... does a further request suspend and cancel off the still running previous request (as seems to happen with download requests)?

The pause for the uploads requested via right click is not possible, but you can see the progress in the bottom progress bar (by clicking the progress circle) or by hovering the circle and can add another batches to the queue.
i.e. works exactly like Send to > Device in MM4

And what happens for download batches? ... do these have pause and resume facility? Or do we have to monitor progress, and only power down, or terminate MM5, when the batch is complete?... can multiple requests be queued without wiping out the queue, as seems to happen at the moment? Is this a bug?

Yes, download batches are persistent after MM5 restart and you have the pause and resume facility. i.e. it is just a bug being fixed as item 10 in https://www.ventismedia.com/mantis/view.php?id=15133 , i.e. will be in 2135