Advanced Duplicate Find & Fix 3.8.2 (2014-01-27)

[Owyn]

See here

[landowin]

Sir, you just made my day.

[Owyn]

[brute cake]

Library->Files to Edit->Dead Links will find all files in library and missing from file system

Search->Advanced:
"Path/Filename" contains "(2)" (Without the quotes) should find the file system duplicates.

owyn: thank you for your suggestion and sorry for taking so long to respond to this thread- for some reason, i never received my email updates notifying me of newer postings...

anyways, I tried to implement your suggestion but have run into two problems. 1) mediamonkey is not reporting any dead links. I believe this is because there aren't any dead links but instead and overabundance of links that need to die.
2) doing an advanced search for files with "(2)" in the path does result in a quite a few results. However, in my prior attempts to rid myself of these horribly pesky duplicates I've ended up with some albums all containing (2) without being duplicates. In other words, a duplicate album was produced and rather than ridding the album(2)'s I deleted the album(original)'s. I suppose 6 of one, half a dozen of another didn't really apply to to me here.

to recap, by the way, here's my problem:

Ok.. so apparently my enthusiasm was a bit premature. I haven't figured anything out it would seem. So I'll try and explain my problem in a different way:
As I mentioned before I have duplicates which have arisen from 2 different scenarios. Scenario 1 is that in the album folder (music is organized in my file system: Artist->Album) I have track.mp3 and track(2).mp3. So two different files, therefore 2 MM library entries. Scenario 2 is that MM is listing Song.mp3 twice as 1.Song and 1.Song, but only one file exists in my file system...

Scenario 1: Track.mp3 & Track(2).mp3 --> Track & Track
Scenario 2: Track.mp3 --> Track & Track

So when creating keep/delete suggestion nodes (using parameters 1.Oldest DateAdded and 2.Lowest SongID), Advanced Duplicate find and fix considers tracks that are file system duplicates (scenario 1) and library duplicates (scenario 2) to be the same. Therefore applying "delete from library and computer" will completely remove albums from scenario 2, rather than just the library duplicates. On the other hand, if I was to just "delete from library only" i would end up with a bunch of duplicates in my file system that simply take up space.

So what I need is a way to discern between these scenarios, so that I can apply "delete from library and computer" to scenario 1, and "delete from library only" to scenario 2.

I hope this helps, it seems to be a pretty confusing issue to explain.
-Brent

thank you,
Brent

[brute cake]

2) doing an advanced search for files with "(2)" in the path does result in a quite a few results. However, in my prior attempts to rid myself of these horribly pesky duplicates I've ended up with some albums all containing (2) without being duplicates. In other words, a duplicate album was produced and rather than ridding the album(2)'s I deleted the album(original)'s. I suppose 6 of one, half a dozen of another didn't really apply to to me here.

Woops, realized that I never finished this thought properly... The problem with this is that since only one copy of the album now exists, deleting the album(2) would cause me to lose the album all together which, obviously, I don't want.

...

[Owyn]

ok.
1)Click "subscribe" at the bottom of this thread to re-subscibe to new posts.
2)This script can not currently find exact duplicates in path. The closest I could find was to check all the fields at the end of the list from Filename through Bitrate and set the tolerances to zero.

I tested this on my approx 46000 item library and found only 24 items. Try this and see if it seems to find the duplicates. While you are doing that I will test some options for cleanup.

[Owyn]

This query can be used in Bex's SQL Viewer script to see if you have songs in your library which point to the same filesystem file.

Code: Select all

SELECT SongPath,COUNT(*) As Dups FROM Songs Group By SongPath Having Dups > 1

There should be no songs selected if there are no duplicate paths.
--------------
SQL-Viewer
viewtopic.php?f=2&t=24841
Let's you run SQL-statements on MM's database exactly as MM itself does it and you'll never get the annoying error:
SQLite Error 1 - no such collation sequence: IUNICODE

[brentiful]

Owyn, thank you so much for your responses.

2)This script can not currently find exact duplicates in path. The closest I could find was to check all the fields at the end of the list from Filename through Bitrate and set the tolerances to zero.

I tested this on my approx 46000 item library and found only 24 items. Try this and see if it seems to find the duplicates. While you are doing that I will test some options for cleanup.

Referring to the above text, I'm not entirely clear on what you are instructing me to do here.. If you could provide me with some more explicit instructions concerning this, I'll give it a shot..

now as far as this goes:

This query can be used in Bex's SQL Viewer script to see if you have songs in your library which point to the same filesystem file.
CODE: SELECT ALL
SELECT SongPath,COUNT(*) As Dups FROM Songs Group By SongPath Having Dups > 1

I executed this query in the SQL viewer (simply copying and pasting "SELECT SongPath,COUNT(*) As Dups FROM Songs Group By SongPath Having Dups > 1" into the query field) and only came up with 15 results as seen in this screen clipping:

Uploaded with ImageShack.us

As you can see in this next screen clipping, these are certainly not the only instances of songs pointing to the same filesystem file. The Air Miami files are all that the SQL viewer found using this query, but the next 2 albums down (Aislers Set and Alaska--just two examples of many) have the same problem, but for some reason do not show up in the SQL viewer. I've hi-lighted examples in the screen shot where i have the same filesystem file showing up twice (which is what i'm trying to get rid of) and circled the air miami album duplicates to hopefully better illustrate this mess. Also you will see that I'm actually suffering from triplicates of some songs in my library. Ahhh!

Uploaded with ImageShack.us

Thank you,
Brent (previously brute cake)

p.s. I signed up for an account with a different user name to avoid having to input the confirmation code each time..

[Owyn]

You actually have 1708 sets of duplicates. The Viewer only displays the first configured number of rows (15 in your case).

Given this I think the simplest thing to do is:
- Backup your library.
- Clear your library
- Add-Rescan your media folders.

The disadvantage of this is you will lose your play history & playlists. These can be recovered later.
The advantage is that it will eliminate all the duplicate file system references and simplify your other cleanups.

An alternative is to use direct SQL commands to remove the duplicates, keeping the lowest id tracks. Still recommend backup before any repairs.
This will need some testing before I can give you the commands to cleanup.

Might want to wait for a second opinion from Bex.

Edit: I am getting the feeling that you moved your media library between devices at some point. Is this correct?

[brentiful]

Edit: I am getting the feeling that you moved your media library between devices at some point. Is this correct?

Haha, basically yes.. This giant headache, believe it or not, has actually arisen out of an attempt to fix an earlier problem that I'd been having. You see, I switched over to mediamonkey from itunes about 2 years ago specifically for its multi-folder monitoring and its robust file tagging capabilities. While great in these regards, trying to manage an ipod with mediamonkey has never been a smooth experience: for some reason, I can never eject my ipod without first exiting out of mediamonkey and then manually ejecting from the taskbar, and also (since i manually manage my ipod library) if I ever accidentally add the same album twice, mediamonkey will simply duplicate it onto my ipod rather than recognizing that its already there and skipping over that album.

2 minor grievances, but enough to eventually cause me to try itunes back out again. During the process, I unwittingly selected an option which caused itunes to begin the process of duplicating my whole library(!). By the time I caught it, everything through B had been duplicated. So, fed up with itunes, I ran back to mediamonkey. Now, I'm still having to deal with my original ipod management issues, as well as this new set of duplicate file problems that has arisen.
Anyways, there have probably been a couple other steps in the process that have led my library to where it dismally is now, but I honestly couldn't tell you exactly what they are (for never being exactly clear on it myself). However, the aforementioned mediaplayer switchover as well as the unintentional deletion of quite a large chunk of music in mediamonkey using the duplicate find and fix script (before I had realized that i had 2 different kinds of duplicates--scenario 1 and scenario 2..) are if-not-completely, then probably mostly to blame.

I'm guessing it was all this that has caused my library to appear as if its moved between devices.


I'll be waiting for bex's opinion on the matter. In the meantime though, I'll begin the process of backing up my library.
Thanks for everything. Hopefully, the headache will be over soon.

-Brent

[Owyn]

Whew.
Let's paraphrase what you said and make sure I understand all the issues.

1)Your media library on your PC is corrupt. You have both duplicate and missing tracks.
2)You may have a good library on your iPod, assuming that you have not transcoded to lower bitrate on the way to your iPod.
3)Your MM database is fubar.

[brentiful]

Whew.
Let's paraphrase what you said and make sure I understand all the issues.

1)Your media library on your PC is c0rrupt. You have both duplicate and missing tracks.
2)You may have a good library on your iPod, assuming that you have not transcoded to lower bitrate on the way to your iPod.
3)Your MM database is fubar.

• 1) no missing tracks.. unless we're defining them differently: Are these dead links we're speaking of, or are we referring to the several gigabytes of music that are no longer due to my duplicate-deleting disaster?
  2) ipod library is a-ok, though I'm waiting to resolve this issue before I finish whittling what makes the cut onto my 80 gigger
  3) and i was hoping this wasn't the case, though i'm starting to think that maybe, it just might be the easiest way to deal with this: Scrap it and start over.

-Brent

[Owyn]

1) no missing tracks.. unless we're defining them differently: Are these dead links we're speaking of, or are we referring to the several gigabytes of music that are no longer due to my duplicate-deleting disaster?

Yep. I assumed you had deleted files from library & computer. Those missing files may exist on either your iPod or an external backup device.

2) ipod library is a-ok, though I'm waiting to resolve this issue before I finish whittling what makes the cut onto my 80 gigger

Does your iPod libary contain a complete copy of your PC media library (ie. the mp3, etc files) before the "duplicate" deletions?

3) and i was hoping this wasn't the case, though i'm starting to think that maybe, it just might be the easiest way to deal with this: Scrap it and start over.

Sh!t happened.

The most important step at the moment is getting a complete set of media files in place. After that you can rebuild your MM database and winnow out duplicate files. You may be able to recover some play history from your iPod but given the shape of your MM database recovering usable stats from the backup database is less likely.

P.S. Which model of iPod is giving you problems with MM?
P.P.S. Realize there may be some confusion in terms.
MM database = Library = SQLite database of meta-data from audio media plus other information not stored in media files (e.g. play history).
Media Library = Folder structures on your hard drives (internal, external or network) which contain your audio files (mp3, aac, etc.).

[Bex]

Brent, check out this:
http://www.mediamonkey.com/forum/viewto ... 26#p138526
It should hopefully get you sorted!

[Owyn]

Yep. That is the alternative SQL I was thinking about with the advantage that Bex's code is tested and working.

You will still have to fix:
- Restore of deleted files
- ,,,(2) duplicates