Logic error after rescanning collection?

This forum is for reporting bugs in MediaMonkey for Windows 4. Note that version 4 is no longer actively maintained as it has been replaced by version 5.

Moderator: Gurus

MMuser2011
Posts: 1308
Joined: Mon Oct 17, 2011 8:28 am
Location: Central Europe

Logic error after rescanning collection?

Post by MMuser2011 »

After rescanning my entire collection, MM needs hours (I mean several hours!) after the last track is scanned until the GUI responds and works as expected.

I assume, after rescanning, some SQL magical work should be done, which needs much more time then necessary.

Could you please tell us which steps are performed after the last track is (re)scanned?
What takes so much time to check, I mean, all the tracks are scannend and therefore exists.
Why does further checks need several hours, especially the list with no more found (moved, deleted, renamed) tracks?

For the record:
MM actually runs on a i9-9900k (8 cores, 16 threads) with 32GB RAM on a very fast M.2 SSD
Magic Nodes v4.3.3 (2018-10-03) last free version SQL Viewer v2.4 (2009-10-25)
ExternalTools v1.4 (2011-05-09) iTunesMonkey 1.0 + Acoustid
Calculate Cover Size v1.7 (2012-10-23) RegExp Find & Replace v4.4.9 (2018-10-03) last free version
Lowlander
Posts: 56491
Joined: Sat Sep 06, 2003 5:53 pm
Location: MediaMonkey 5

Re: Logic error after rescanning collection?

Post by Lowlander »

Which version of MediaMonkey?

Do you have auto volume analyze enabled?
MMuser2011
Posts: 1308
Joined: Mon Oct 17, 2011 8:28 am
Location: Central Europe

Re: Logic error after rescanning collection?

Post by MMuser2011 »

Version: 4.1.24.1883
"Auto volume analyze" is disabled/not checked
Magic Nodes v4.3.3 (2018-10-03) last free version SQL Viewer v2.4 (2009-10-25)
ExternalTools v1.4 (2011-05-09) iTunesMonkey 1.0 + Acoustid
Calculate Cover Size v1.7 (2012-10-23) RegExp Find & Replace v4.4.9 (2018-10-03) last free version
Lowlander
Posts: 56491
Joined: Sat Sep 06, 2003 5:53 pm
Location: MediaMonkey 5

Re: Logic error after rescanning collection?

Post by Lowlander »

A debug log (step 4b) will show what's happening: https://www.mediamonkey.com/forum/viewt ... p?f=7&t=69 You can attach it to a Support Ticket for the developers to analyze.
MMuser2011
Posts: 1308
Joined: Mon Oct 17, 2011 8:28 am
Location: Central Europe

Re: Logic error after rescanning collection?

Post by MMuser2011 »

First of all: Thank you for the MM debug version and the inside informations.

The latest rescan for all my songs took 44 minutes.
(from start scanning until the point where MM seems to be frozen with the last "added" song shown in the taskbar,
that's the point, where it takes hours until MM returns to normal working state***)

The following should be noted:
Every step you do works fine. At least for a few hundred or a thousand tracks.
But please think bigger. What about hundreds of thousands or even millions of tracks?
Thanks to the DebugView log I would like to suggest some improvements:

Example:
For every single track you query the MM.DB with
SELECT Songs.* FROM Songs WHERE Songs.IDMedia=? and Songs.SongsPath=?
Even on a very fast system this takes - according to the DebugView-Output -
about 450-600microseconds, or 0.5 miliseconds or 0.0005 seconds.

Again: This is not a problem for a few hundred songs. But for millions?
If you calculate 1 Million tracks you need 1'000'000 x 0.0005 seconds = 500 Seconds = 8,3 Minutes only for this query

And this query is only one of many processed for every track.

I suggest to rethink this process.
Step #1:
Assuming that most of us has some structure for the songfiles,
we could think about something like a folder or an album.
Why not read a whole folder from disk with something like "Directory.EnumerateFiles"?
You make only one call per folder and get all the full names (including paths) for the files in the specified directory.
Then you do the same with only one query with the specified foldername in the MM.DB with
SELECT Songs.* FROM Songs WHERE Songs.IDMedia=? and Songs.SongsPath=FolderName

After matching the songs in memory you have two kind of entries
a) the ones found in DB = all this songs already exists in DB
b) the ones not found in DB = new or changed ones

For b) you have to scan every new file to get the additional tag information. The way it works today.
For a) you only have to scan the files with changed "last modified" timestamp (according to the user settings).

BTW: I don't know which programming language do you use. But with the help of something like this
https://sourceforge.net/projects/fastfileinfo/
original source was: https://www.codeproject.com/Articles/38 ... Enumerator
you can speed up things dramatically.


Step #2:
CREATE TEMP TABLE ScannedSongs_2 ( IDSong INTEGER )
DB exec SQL: INSERT INTO ScannedSongs_2 (IDSong) VALUES (?)

I'm not sure if this table is created in the MM.DB or just as memory table.
Anyway: I would suggest to use transactions for all the INSERT INTO's and not execute it for every single track.
(The same way you do it already to UPDATE Songs SET SongPath=?,Artist=?....)


Step #3:
AddToDatabaseRec: Number of cover candidates: 0
AddToDatabaseRec: Going to save covers

I'm not sure what exactly you are doing to decide to get 0 cover candidates.
In my case I have deactivated the setting "search for covers in the same directory".
So it should not search at all for covers especially not with an additional "file search"- or "cover save"-call.


Step #4:
Prgrs: Updating previous level from: 0 by 10
Prgrs: Updating previous level from: 0.3 by 0.7
Prgrs: Updating previous level from: 510 by 10
Prgrs: Creating a new level.
Prgrs: Deleting a level.

I'm not sure what exactly you are doing while "Updating previous level"
If it is about the volume level per Track or Album:
In my case I have deactivated all the volume leveling settings.
This is confirmed by several "TDevices.disableDiscVolume" log entries.


***
Final words to the endless last step (after rescanning the songs, until the GUI responds again):
MM is actually writing endless lines of
Thread '_ManageTransactions' has started/has finished with 4 or 5 lines of
DB open SQL: SELECT * FORM Songs WHERE Songs.ID=?

5 lines needs about a second.
I assume, you will do that for all tracks.
1'000'000 : 5 = 200'000 seconds = 3'333 minutes = 55,5 hours...

Why just using 2 Threads from all 32 available?
Do you really write only 4 or 5 songs with a single transaction?
Magic Nodes v4.3.3 (2018-10-03) last free version SQL Viewer v2.4 (2009-10-25)
ExternalTools v1.4 (2011-05-09) iTunesMonkey 1.0 + Acoustid
Calculate Cover Size v1.7 (2012-10-23) RegExp Find & Replace v4.4.9 (2018-10-03) last free version
Erwin Hanzl
Posts: 1189
Joined: Tue Jun 13, 2017 8:47 am
Location: Vienna

Re: Logic error after rescanning collection?

Post by Erwin Hanzl »

Gelöscht
Last edited by Erwin Hanzl on Sun Jun 23, 2019 8:17 am, edited 1 time in total.
MMW 4.1.31.1919 Gold-Standardinstallation
Peke
Posts: 17457
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: Logic error after rescanning collection?

Post by Peke »

Hi,
MMuser2011 wrote: Fri Apr 12, 2019 4:23 am Why just using 2 Threads from all 32 available?
Can you try same scan using clean MM5 Portable. I tested today 30k of tracks saved on SATA3 SSD and AMD 2700x CPU on M.2 Samsung 970 Pro system drive and scan finished within 12 Minutes. All 16 threads were evenly used.

My System Info:
Image
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
MMuser2011
Posts: 1308
Joined: Mon Oct 17, 2011 8:28 am
Location: Central Europe

Re: Logic error after rescanning collection?

Post by MMuser2011 »

Peke wrote: Tue May 21, 2019 1:00 pmCan you try same scan using clean MM5 Portable...All 16 threads were evenly used.
Done.

MM5 portable (running the MM.DB on a Samsung SSD 850 PRO 512GB SATA SSD) needs 41 Minute for a complete rescan. So MM5 is slightly faster.
What exactly do you have changed in this process?

And you are right, MM5 uses all 16 Threads (not 32, as I mistakenly wrote based on an outdated gadget. My i9-9900K has 8 Cores, 16 Threads).
Image

The main problem - the GUI freezes after this 41 MM5 minutes - remains unchanged.
One of the 16 threads run constantly at 90%, the other 15 threads shows about 0% to 5% load/utilization.
Magic Nodes v4.3.3 (2018-10-03) last free version SQL Viewer v2.4 (2009-10-25)
ExternalTools v1.4 (2011-05-09) iTunesMonkey 1.0 + Acoustid
Calculate Cover Size v1.7 (2012-10-23) RegExp Find & Replace v4.4.9 (2018-10-03) last free version
Peke
Posts: 17457
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: Logic error after rescanning collection?

Post by Peke »

Hi,
That needs optimization. I would be great to fetch Video during the scan and MM5 Debug log showing where problem lies and move some work off main thread. I do not observe your freeze, but I see your point and that should certainly be corrected.

MM5 is simply better multi thread optimized, especially that MMW takes root in 15+ years of developing when things looked/worked differently.
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
MMuser2011
Posts: 1308
Joined: Mon Oct 17, 2011 8:28 am
Location: Central Europe

Re: Logic error after rescanning collection?

Post by MMuser2011 »

You ask for a 41 minutes Video?

Unfortunately, the debugfile gets too large, MM crashes after a certain size.
Magic Nodes v4.3.3 (2018-10-03) last free version SQL Viewer v2.4 (2009-10-25)
ExternalTools v1.4 (2011-05-09) iTunesMonkey 1.0 + Acoustid
Calculate Cover Size v1.7 (2012-10-23) RegExp Find & Replace v4.4.9 (2018-10-03) last free version
Peke
Posts: 17457
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: Logic error after rescanning collection?

Post by Peke »

Hi,
Not all 41 Minute only few last minutes.

What do you mean LOG gets too Large > 2GB?
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
MMuser2011
Posts: 1308
Joined: Mon Oct 17, 2011 8:28 am
Location: Central Europe

Re: Logic error after rescanning collection?

Post by MMuser2011 »

DebugView can't list more then 6'646'529 lines in it's listview :wink:

Image

@Peke You got a PM

There you can see what happens after the last track was scanned. The last 150 lines repeats "endlessly".
Magic Nodes v4.3.3 (2018-10-03) last free version SQL Viewer v2.4 (2009-10-25)
ExternalTools v1.4 (2011-05-09) iTunesMonkey 1.0 + Acoustid
Calculate Cover Size v1.7 (2012-10-23) RegExp Find & Replace v4.4.9 (2018-10-03) last free version
Post Reply