What Is A Duplicate?

OK. Obviously, a duplicate is one thing that you consider to be a copy of another thing.

However, it has been my experience, based on a countless number of email queries, that there are many kinds of duplicates. Indeed, the perception of what a duplicate is can vary from user to user! I will try to sort out the different types of duplicates below.

But first, a few definitions.

• "Music file", "audio file", "sound file", or just plain "file":

A music file is a computer file that contains sound data. These are the things you see in the Finder, or download from the iTunes Music Store, or rip from a CD, and so on.

• "Music track", "audio track", "iTunes track", "track":

A track in iTunes is a representation of an audio file in iTunes—an alias, if you will. These are not the same as files! When you import or add a file to iTunes, the actual file does not get put into iTunes. Believe it or not, this is a misconception some users have.

When you import, convert, purchase, or (possibly) add a file to iTunes, iTunes creates a new entry in its database for the file and creates a new track in the main library. The file is also assigned a database id in the iTunes database, completely unique for that file.

Files are in the Finder, tracks are in iTunes. The unique database id for the file is listed as the "Track ID" in the database. The path to the file is listed as the "Location",
which is how iTunes knows where the audio file is.

So basically, when you activate a track in iTunes, iTunes locates the audio file it represents, processes it, and "plays" it.

• iTunes Library:

The iTunes library, or main library, is the topmost playlist in the iTunes Source list where every track whose file you have added to iTunes is listed as a track. (I will not use library to refer to the "iTunes Music folder" where files are stored, and neither should you, see below)

A misconception some users have is that the library is just another playlist that happens to be permanent. While this is true to some extent, you should consider the library as the place where you can organize and access tracks to edit their tags. While you are certainly able to listen to tracks from the library, ideally, you should be playing them from a playlist.

• iTunes playlist:

A playlist is a list of tracks you have created from the library. Presumably, you store your tracks in the library and play them from a playlist.

Think of it like this: in the old days, music aficionados, such as myself, kept shelves full of CDs (or vinyl records) in their house. I would piddle with my music collection on the shelves, organizing it by genre or artist or album name, or what have you. But, when I wanted to play a recording, I removed it from its shelf and brought it to the CD player (or phonograph) and played it; it was moved from its storage area to the playing area.

Likewise, in iTunes, the library (storage area) is used to maintain track tags, organize albums, and so on. The playlist (playing area) is used to create a list of tracks you want to listen to.

When you copy a library track to a playlist (or a Smart Playlist automatically), iTunes knows that you are referring to a single particular audio file. Each track added to the playlist has the same database id as its counterpart in the library. You have only created another reference to the same file. But they are not duplicates, just references to the same audio file.

• Where Do Files Go?

iTunes tracks, of course, are in the iTunes library and any playlists you have distributed them to. Where are the corresponding files?

iTunes, by default, stores files you have imported from CD, purchased from the iTunes Music Store, downloaded as Podcasts, or converted from other iTunes tracks into the default iTunes Music folder. This is the topmost folder wherein those audio files (imported, purchased, or converted) are located. When iTunes is first installed, it creates a folder called "iTunes Music" in the username > Music > iTunes folder. However, because you can change the location of this folder in iTunes Preferences, I prefer to call this folder the designated Music folder. This is not official Apple nomenclature, by the way.

Another factor in determining where files are located is the "Copy files to iTunes Music folder when adding to library" Preferences setting. Note that the operative word here is "adding".

The command (verb?) add is a separate process from import and convert. Technically, a file that already exists in the Finder is added to iTunes, either by using the "Add to Library..." File menu command and choosing a file (or files), or by drag-and-dropping an audio file (or files) onto iTunes.

If the "Copy files to iTunes Music folder when adding to library" Preference is checked then added files are copied to the designated Music folder. The original file is not moved, however iTunes only knows about the file it copied to the designated Music folder. Additionally, if you have also checked the "Keep iTunes Music folder organized" Preference setting checked, then files will be organized into Artist and Album folders based on their corresponding tags. iTunes then creates a database entry for the file it has copied to the designated Music folder.

If the "Copy files to iTunes Music folder when adding to library" Preference is unchecked then the added files remain where they are and iTunes creates a database entry for the file where it currently exists.

Thus, depending on how you have your iTunes Preferences set up, your ripped CD files, converted tracks' files, and purchased music could end up in one place (always in the designated Music folder) and other files could remain in another.

In any case:

iTunes will not allow a file to be re-added if it is already aware of the same file in its database.

iTunes will not allow a file to be re-added if it detects its Song Name, Artist, and Album name is already in its database.

Case #1: Copying the Same Library Track to a Playlist Two or More Times

There is a possibility that you could have tracks which refer to the same audio file listed multiple times in a playlist.

It can be done! Try dragging a library track to a playlist. Now, drag it again. And again. And again. When you look at the playlist, you have four identical tracks! Each of the tracks will have the same database id, so as far as iTunes is concerned they all point to the same audio file. You can remove all but one of these types of duplicate tracks using the script Remove Duplicate Tracks From Playlist. The script looks for duplicate occurrences of a track's database id.

By the way, you cannot create a duplicate library track from the same audio file. iTunes will not allow you to re-add a file to the library if it already has an entry to that particular file in its database.

Case #2: The "Original" and "Duplicate" Myth

I wish I had a penny for every email I receive which queries: "How can I remove all my duplicate tracks and keep the originals?" I would be the Wealthiest Fellow such that this site's Amazon, Kagi, and PayPal payboxes would be completely unnecessary.

This question suggests that the poser has a grand misconception about tracks and files. In fact, I am hard-pressed to even comprehend the question. iTunes barely has a clue as to what the "original" track is. iTunes has even less of a clue as to what the "duplicate" track/s is/are.

Audio files are not like text files. You can't examine them to see if two are the same. (OK, you can compare tags and kind and size information, but even this can produce mixed results, as users of the iTunes "Show Duplicate Songs" command generally atest.) Tracks represent sound files. So, you have to listen to them. This seems odd to computer users because we are used to applications that can sniff out the similarities of files very quickly. But when you get right down to it, despite the text-based nature of audio file ID3 tags, only a device with ears can be the final arbiter of what a duplicate sounds like compared to the original. You have ears, I think. Right?

Some have suggested that an AppleScript could be written to "eliminate" the duplicates based on such things as last played date, play count, date added, and such. In my view, this still borders on the arbitrary. However, I have written various scripts which attempt to locate duplicates based on various data; these are about as accurate, more or less, as the iTunes "Show Duplicate Songs" command. Ultimately, you still have to listen to tracks to determine which can stay or go.

Also, I have written an application called Dupin that will locate groups of duplicate tracks based on criteria you select, and then determine the "keeper" among each group according to a set of filter options. You can then purge the "non-keepers" from iTunes. Note, however, that this still requires your participation; Dupin does not automatically delete anything. Even so, no piece of software really knows what you think a sound file duplicate is.

Case #3: Ooops, I Re-Ripped It

I have ripped a CD two or more times. I forgot that I ripped it a first time, so eleven files were imported and added to iTunes which I already had.

If the tracks of the CD you are ripping have the same Song Name, Artist, and Album information as tracks you have already ripped, iTunes will detect this and ask if you want to replace the current iTunes tracks with new tracks of the files you are about to rip. This is easy for iTunes to do if you have set up your designated Music folder to organize files by Artist folder > Album folder > Song Name file. It just compares the tag data of the CD tracks to known locations in its database file. (You can use the script Import on CD Insert, which will pre-empt this iTunes feature.) If you select not to replace the files, then when new files are ripped from a CD, the Finder will name the new files appending a number after each file name, like so:

Ooops! I ripped 'em again!

...and iTunes, perceiving these as new and different files, will add them to its library. Voila! Duplicates!

Many people have suggested that a script could be written that detects and Trashes the files with the affixed number. My problem with this is: What if the second or third or whatever rip is The Keeper? Again, ears come in handy.

Here's another instance of re-ripping duplicates. Perhaps you have changed the Song Name, Artist, and Album information of either the CD tracks or the tracks in iTunes which you had previously ripped from that CD. For example, you have edited the Album name "Woodstock 99 (Disc 1)" to just "Woodstock 99". In this case, because the Album name has been changed, iTunes will assume that the CD tracks are different from any in its library and will import them without warning you that they may be duplicates.

Case #4: Greatest Hits

Similar to the previous Case, you may have ripped a CD that contains tracks you already have in your iTunes library. However, because the name of the CD Album or Artist is different, the tracks will be imported as duplicates. In this situation, the files will probably be stored in different Artist or Album folders. For example, the location of the file "Witchy Woman" by the Eagles from their album "Eagles" will be stored in Eagles > Eagles > Witchy Woman.m4a. But the same song ripped from "The Eagles - Their Greatest Hits" will be stored in Eagles > The Eagles - Their Greatest Hits > Witchy Woman.m4a. Voila! Duplicate!

Which one do you want to keep? If it was me, I'd want to keep both albums intact, despite the fact that the same songs exist on different albums. But, again, it will be you (and your ears) who decides which can stay or go.

Case #5: Ripped In A Different Format

You ripped a CD or added tracks in a different format than the ones extant in iTunes. I usually rip my CDs in a standard format, usually AAC. However, my daughter has an audio player which only accepts MP3 files. This means I either have to convert the extant AAC file to MP3 (not likely! due to audio degradation, and I'm raising my daughter to be descriminating as far as audio quality) or re-rip the CD tracks as MP3 files. If so, I get duplicate MP3 files of AACs I have already ripped. And because the file names are different (files end with ".mp3" and ".m4a") I get duplicate tracks.

You could sort your library by "Kind" and delete the ones you don't want; you could set up a Smart Playlist to exclude one or the other file kind; use iTunes "Show Duplicate Songs" menu command and segregate from there. As usual, ears work good.

Case #6: I Got It From A Friend

Your Buddy has sent you a CD chock full of his favorite tracks. (I do not condone file sharing, but it happens. Perhaps your Mother has sent you a CD of your late Father's favorite Fats Waller tunes, which Really Happened to me.). So naturally, you want to add them to iTunes. Oops! I already had "Carolina Shout", but it ended up in the "Unknown Artist" and "Unknown Album" folder. Still, the thing shows up in iTunes against your legitimate copy of "Carolina Shout".

Like the Cases above, the file was placed in an unexpected place. Again, your ears (and your sentiments) dictate what happens next.

Case #7: Re-Adding From Backup

This is a tough scenario to describe. Since I have no way of knowing how you have backed-up your music files, Anything can happen. Let's consider some factors.

Generally speaking, when you re-import audio files from a back-up location (external drive, CD, or DVD) and the tags of the backup files correspond with tags of extant iTunes library tracks, the backup files will not import. This is because iTunes tries to avoid re-importing files it is already aware of.

On the other hand, if the tag info of your backup files does not correspond to the info of extant files, the backup files will be added, and could produce duplicates.

I suppose the safest thing to do is make sure your tag info is set in stone before making backups. Any changes to iTunes library tracks or to the backup files can possibly result in duplicates.

Another backup strategy is to make regular backups of your entire designated Music folder--the one pointed to in iTunes Preferences > Advanced> General > iTunes Music folder location--along with the iTunes database file. The iTunes database file, most recently named "iTunes Library", is located in username > Music > iTunes. When you want to restore from backup, Quit iTunes and replace the current username > Music > iTunes > iTunes Library file with your backup copy (which ensures the playlists remain intact), and then replace your designated Music folder with the one you have backed up. Restart iTunes.

User Reporting

The Cases listed below have been submitted by Correspondents. I cannot necessarily verify their integrity.

Case #8: Consolidating the Library

Thanks to my friend Lita for pointing this out.

You may have your audio files spread out All Over The Place, and thus, to bring them all together Apple recommends you use the "Advanced" menu command "Consolidate Library". "Any song files that are located outside the iTunes Music folder remain in their original location; copies of them are placed in your iTunes Music folder." (As mentioned above, the "iTunes Music folder" is what I am calling the designated Music folder.)

What happens is that "Consolidate Library" makes copies of all music files 1) whose tracks are in the iTunes library and 2) that aren't in the designated Music folder; and changes the database index to reflect the new location. If you run "Consolidate Library" again after changing the location of the designated Music folder you will get duplicate files. This will also happen even if you manually move the files to the designated Music folder because iTunes still thinks they are in the original location.

Case #9: Using Another App to Tag Your Tracks

Thanks to Correspondent Tod W. for this one.

There are some very fine applications available that can assist you with applying ID3 tags to your audio files (MP3Rage, MPFreaker, AutoTag, et al). Some work well with iTunes and some do not. It is beyond the scope of this article to compare their effectiveness in this regard. However, depending on which audio files you direct them to tag, for instance backups rather than those in your designated Music folder, you may be creating tags which iTunes will later be unable to reconcile (for whatever reason). Those files may be considered "new" files when it comes time to re-add them. You guessed it: duplicates.