LastFM automated duplicate scrobble deletion script

lastfm-automated-remove-duplicates.md

Why would I need this?

Your scrobbler might have decided to scrobble every song hundreds of times, and you can't really remove those scrobbles efficiently. Or you might have accidentally installed multiple scrobbler extensions at the same time - wondering why multiple scrobbles appear for every song played at a time - and you want to clear them after finding the issue.

Using this script still doesn't necessarily make the process quick since Last.fm only displays a limited number of scrobbles that can be removed on each page of your library. However unlike the implementation of @sk22 and its forks, this UserScript, which is derived from those scripts, is run once. The rest of the process is automated and the script will stop at the page you have set using the prompt.

Installation

Prerequisites

You will need some form of UserScript interpreter/injector plugin and a compatible browser.

• For Google Chrome and Chromium-based browsers (e.g. Vivaldi, Opera or Brave), you need the Tampermonkey plugin. Just click on the link or search for it yourself on the Chrome Web Store and add the extension.
• Currently not supported: For Firefox, you also need the Tampermonkey plugin (or alternatively Greasemonkey). Just click on one of the desired links or search for it yourself on the Firefox Browser Add-Ons page.

How to install the UserScript

Just click on the "Raw" button in the upper right-hand corner of the script file or by clicking here. Your extension should open an installation window where you confirm the installation of the UserScript. If that is not working, try to add the script manually to your extension by copying and pasting it.

Using the script - step by step

• Open your Last.fm account's library while being logged in (https://www.last.fm/user/_/library). Navigate to the page from which you want to start removing duplicates by clicking through the pages at the bottom of the page or by entering the page number in the URL (e.g. https://www.last.fm/user/_/library?page=123).
  • Alternatively, you can choose a specific date range instead and navigate to the last or desired page.
• Manually reload the page if the button doesn't appear.
• You have the option to change the behaviour of the comparison process. By unchecking the checkbox, you can make the script ignore timestamps and compare over n titles. You'll be prompted to enter a number after clicking the button.
• Click the button and enter the page number at which you want the script to stop.
  • If you have unchecked the checkbox, you will be prompted to enter a number for how many tracks you want to compare.
• Now enjoy the purge :)
  • You can continue with other tasks on your PC. Just leave the window active in the background. I advise you to run the script in a separate window while doing other work. Using the same browser in another window is also fine. Avoid activating another tab, though.
  • It takes approximately 5 minutes to go through 100 pages (in my own case).
  • If you want to stop the script, you can click the Cancel button on the left-hand side.
• When it reaches the page you have set at the prompt, the script will automatically stop.

Updating the script

You can set up the script to auto-update in the settings of the UserScript. Open up the plugin settings and select the "LastFM automated duplicate scrobble deletion script." Then, go to the script settings in the upper left-hand corner, check the "Search for updates" checkbox and save.

Alternatively, you can also update it manually by following the installation instructions above or searching for updates in the plugin overview.

Possible future updates

• Ability to differentiate within days, hours or minutes (implemented on a different version here: https://gist.github.com/olphil99/161cd24cba9db74fc945f3ebf167a54a by @olphil99, currently not working though)
  • Alternatively, starting from version 1.2.1, the script can now remove duplicates that share the same timestamp.

Done

• Adding a button on the last.fm history page to initiate the script instead of activating and deactivating the Userscript.
  • added in v1.3
• Implementing a "stop at page x" dialogue window.
  • added in v1.3
• Implementing a "check x tracks for duplicates" dialogue window instead of editing "num" in the script
  • added in v1.3
• Eliminate the need to manually edit the script by incorporating a checkbox or another method for configuring options.
  • added in v1.4

---

Thank-yous

Based on https://gist.github.com/sk22/39cc280840f9d82df574c15d6eda6629#gistcomment-3046698 Thanks to previous contributors @CennoxX, @mattsson, @gms8994, @huw and @sk22 Thanks to new contributions from @Eiron

Feel free to post any problems, fixes, improvements and feedback. I will try to help in any form. To anyone having more experience in Javascript or OpenUserJS/UserScript: your help is also needed to improve the script by implementing the above-mentioned possible future updates as I do not have the necessary skills to contribute.

lastfm-automated-remove-duplicates.user.js

// ==UserScript==
// @name         LastFM automated duplicate scrobble deletion script
// @namespace    https://gist.github.com/muhdiboy
// @version      1.5.0
// @downloadURL  https://gist.github.com/muhdiboy/a293cbff355af750e3b8f45ec816d1f1/raw/lastfm-automated-remove-duplicates.user.js
// @updateURL    https://gist.github.com/muhdiboy/a293cbff355af750e3b8f45ec816d1f1/raw/lastfm-automated-remove-duplicates.user.js
// @description  Based on https://gist.github.com/sk22/39cc280840f9d82df574c15d6eda6629#gistcomment-3046698, thanks to previous contributors (@CennoxX, @mattsson, @gms8994, @huw and @sk22) and new contributors (@Eiron).
// @author       muhdiboy
// @match        https://www.last.fm/*user/*
// @icon         data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==
// @grant        none
// ==/UserScript==

(function() {
    // Variables
    let runScript = localStorage.getItem('runScript') === 'true';
    let rmOnlyRealDup = localStorage.getItem('rmOnlyRealDup') !== 'false'; // Set checkbox to checked/true as default
    let numTracks = localStorage.getItem('numTracks') || "5";
    let stopAtPage = localStorage.getItem('stopAtPage') || "0";
    let currentPage = extractQueryParam('page', window.location.href) || 1;

    // Update currentPage when a click event occurs on the page
    window.addEventListener('click', function () {
        currentPage = extractQueryParam('page', window.location.href) || 1;
    });

    // Functions
    let pageURLCheckInterval = setInterval (function () {
        // Check the URL at regular intervals to add or remove buttons
        if ( this.lastPathStr !== location.pathname || this.lastQueryStr !== location.search) {
            this.lastPathStr = location.pathname;
            this.lastQueryStr = location.search;
            if (shouldMatchPage()) createButtons(); else removeButtons();
        }
    }, 125);

    function replaceQueryParam(param, newval, search) {
        // Function to replace a query parameter in the URL
        const regex = new RegExp("([?;&])" + param + "[^&;]*[;&]?");
        const query = search.replace(regex, "$1").replace(/&$/, '');
        return (query.length > 2 ? query + "&" : "?") + (newval ? param + "=" + newval : '');
    }

    function extractQueryParam(name, url) {
        // Function to extract a query parameter from the URL
        if (!url) url = window.location.href;
        name = name.replace(/[\[]/, "\\[").replace(/[\]]/, "\\]");
        const regexS = "[\\?&]" + name + "=([^&#]*)";
        const regex = new RegExp(regexS);
        const results = regex.exec(url);
        return results == null ? null : results[1];
    }

    function shouldMatchPage() {
        // Function to determine if the current page should be matched
        const url = window.location.href;
        const excludeUrlRegex = /^https:\/\/www\.last\.fm\/.*user\/.*\/library\/(albums|artists|tracks|music).*/;
        const matchUrlRegex = /^https:\/\/www\.last\.fm\/.*user\/.*\/library.*/;
        return matchUrlRegex.test(url) && !excludeUrlRegex.test(url);
    }

    function removeButtons() {
        // Function to remove all buttons from the page
        const cancelButton = document.getElementById('cancelButton');
        const buttonContainer = document.getElementById('buttonContainer');

        if (cancelButton) cancelButton.remove();
        if (buttonContainer) buttonContainer.remove();
    }

    // Buttons
    function createButtons() {
        // Create the cancel button to stop the script
        const cancelButton = document.createElement('button');
        cancelButton.id = 'cancelButton';
        cancelButton.innerHTML = 'Cancel<br>Script';
        cancelButton.style.position = 'fixed';
        cancelButton.style.bottom = '50%';
        cancelButton.style.left = '0';
        cancelButton.style.zIndex = '9999';
        cancelButton.style.fontFamily = 'monospace';
        cancelButton.style.padding = '3px';
        cancelButton.style.border = 'dashed';
        if (runScript) {
            cancelButton.style.color = 'white';
            cancelButton.style.background = 'red';
            cancelButton.style.borderColor = 'firebrick';
            cancelButton.addEventListener('click', function() {
                localStorage.setItem('runScript', 'false');
                localStorage.removeItem('savedUrl');
                location.reload();
            });
        } else {
            cancelButton.style.color = 'gray';
            cancelButton.style.background = 'lightgray';
            cancelButton.style.borderColor = 'lightgray';
            cancelButton.style.cursor = 'default';
        }
        document.body.appendChild(cancelButton);

        if (!runScript) {
            // Create the main button to trigger the script
            const runButton = document.createElement('button');
            runButton.textContent = 'Remove Duplicates';
            runButton.style.position = 'relative';
            runButton.style.display = 'block';
            runButton.style.marginBottom = '5px';
            runButton.style.fontFamily = 'sans-serif';
            runButton.style.fontSize = 'initial';
            runButton.style.border = 'outset';
            runButton.style.borderColor = 'crimson';
            runButton.style.color = 'white';
            runButton.style.background = 'firebrick';
            runButton.style.padding = '5px';
            runButton.addEventListener('click', function () {
                let stopLoop = false;

                while (!stopLoop) {
                    const inputStopAtPage = prompt("Stop at which page?", stopAtPage);
                    if (inputStopAtPage === null) return;
                    if (!/^\d+$/.test(inputStopAtPage)) {
                        alert("Invalid input. Please enter a valid number.");
                        continue;
                    }
                    if (inputStopAtPage > currentPage && !(currentPage === 0 && inputStopAtPage === 1)) {
                        alert("Please enter a page less than or equal to your current page: " + currentPage);
                        continue;
                    }
                    localStorage.setItem('stopAtPage', inputStopAtPage);
                    stopLoop = true;
                }

                if (!rmOnlyRealDup) {
                    stopLoop = false;
                    while (!stopLoop) {
                        const inputNumTracks = prompt("Enter the number of tracks to check for duplicates\n(between 2 and 50):", numTracks);
                        if (inputNumTracks === null) return;
                        if (!/^\d+$/.test(inputNumTracks) || inputNumTracks < 2 || inputNumTracks > 50) {
                            alert("Invalid input. Please enter a valid number between 2 and 50.");
                            continue;
                        }
                        localStorage.setItem('numTracks', inputNumTracks);
                        stopLoop = true;
                    }
                }

                localStorage.setItem('runScript', 'true');
                location.reload();
            });

            // Create the checkbox element
            const rmOnlyRealDupCheckbox = document.createElement('input');
            rmOnlyRealDupCheckbox.type = 'checkbox';
            rmOnlyRealDupCheckbox.id = 'rmOnlyRealDupCheckbox';
            rmOnlyRealDupCheckbox.style.position = 'relative';
            rmOnlyRealDupCheckbox.style.marginLeft = '10px';
            rmOnlyRealDupCheckbox.checked = rmOnlyRealDup;
            rmOnlyRealDupCheckbox.addEventListener('change', function () {
                localStorage.setItem('rmOnlyRealDup', rmOnlyRealDupCheckbox.checked);
            });

            // Create the label for the checkbox
            const rmOnlyRealDupCheckboxLabel = document.createElement('label');
            rmOnlyRealDupCheckboxLabel.style.position = 'relative';
            rmOnlyRealDupCheckboxLabel.style.marginLeft = '5px';
            rmOnlyRealDupCheckboxLabel.style.fontFamily = 'sans-serif';
            rmOnlyRealDupCheckboxLabel.style.fontSize = 'medium';
            rmOnlyRealDupCheckboxLabel.setAttribute('for', 'rmOnlyRealDupCheckbox');
            rmOnlyRealDupCheckboxLabel.textContent = 'Only compare real duplicates?';

            // Create a container element to hold the main button and checkbox
            const buttonContainer = document.createElement('div');
            buttonContainer.id = 'buttonContainer';
            buttonContainer.style.position = 'absolute';
            buttonContainer.style.top = '370px';
            buttonContainer.style.left = '50%';
            buttonContainer.style.transform = 'translate(-50%, -50%)';
            buttonContainer.appendChild(runButton);
            buttonContainer.appendChild(rmOnlyRealDupCheckboxLabel);
            buttonContainer.appendChild(rmOnlyRealDupCheckbox);
            const metaElement = document.querySelector('.content-top-has-nav');
            metaElement.insertAdjacentElement('afterend', buttonContainer);
        }
    }

    // Automated duplicate scrobble deletion and navigation logic
    if (shouldMatchPage()) {
        window.addEventListener('load', function () {
            // Event listener for when the page has finished loading
            if (runScript) {
                let found = 0;
                const initialRefreshTimeout = 5000; // Interval for reloading the page
                let refreshTimeout = parseInt(localStorage.getItem('refreshTimeout')) || initialRefreshTimeout;
                const sections = Array.from(document.getElementsByTagName("tbody"));
                const savedUrl = localStorage.getItem('savedUrl') || window.location.href;

                // If the URL contains "delete", go back to last functioning site
                if (window.location.toString().includes("delete")) window.location.href = savedUrl;

                // If the current page doesn't contain tracks, go back to last functioning site (useful to detect rate-limiting page)
                if (!document.querySelector('.chartlist')) {
                    setTimeout(function() {
                        window.location.href = savedUrl;
                    }, refreshTimeout);
                    localStorage.setItem('refreshTimeout', refreshTimeout + 5000);
                    return;

                } else {
                    refreshTimeout = initialRefreshTimeout;
                    localStorage.setItem('refreshTimeout', initialRefreshTimeout);
                    localStorage.setItem('savedUrl', window.location.href);
                }

                sections.forEach(function (section) {
                    // Loop through each section
                    const els = Array.from(section.rows);
                    const names = els.map(function (el) {
                        const nmEl = el.querySelector('.chartlist-name');
                        const artEl = el.querySelector('.chartlist-artist');
                        if (rmOnlyRealDup) {
                            const tstEl = el.querySelector('.chartlist-timestamp');
                            // Construct the name string including track name, artist and timestamp
                            return nmEl && artEl && tstEl && nmEl.textContent.replace(/\s+/g, ' ').trim() + ':' + artEl.textContent.replace(/\s+/g, ' ').trim() + ':' + tstEl.textContent.replace(/\s+/g, ' ').trim();
                        } else {
                            // Construct the name string including only track name and artist
                            return nmEl && artEl && nmEl.textContent.replace(/\s+/g, ' ').trim() + ':' + artEl.textContent.replace(/\s+/g, ' ').trim();
                        }
                    });

                    names.forEach(function (name, i) {
                        // Loop through each name in the section
                        if (!names.slice(i + 1, i + 1 + parseInt(numTracks)).includes(name)) return;
                        // Check if the current name has duplicates within the specified range
                        const delBtn = els[i].querySelector('[data-ajax-form-sets-state="deleted"]');
                        // If a delete button is found, click it and increment the counter
                        if (delBtn) { delBtn.click(); found++; }
                    });
                });

                if (found > 0) {
                    // If duplicates were found, reload the page after refreshTimeout
                    setTimeout(function() {
                        location.reload();
                    }, refreshTimeout);
                    return;
                }

                if (currentPage <= stopAtPage) {
                    // Stop the Script and message user
                    localStorage.setItem('runScript', 'false');
                    localStorage.removeItem('savedUrl');
                    alert("LastFM duplicate deletion complete.");
                    location.reload();
                }

                // Go up one page
                else window.location.href = window.location.pathname + replaceQueryParam('page', extractQueryParam('page', window.location.href) - 1, window.location.search);
            }
        });
    }
})();

@Jeboose
Can you retest with the newest version? If you have a different issue, please retest on a single page where you are sure multiple duplicates exist and report your findings. I will try to help as possible.

Just to be sure, if you didn't know the feature: the script is differentiating duplicates, ones that are scrobbled at exactly the same time (to the minute) and ones where you can choose the amount to be checked. First one is the true duplicate, enabled by the checkmark. Second one is active when the checkmark is unchecked. When unchecked, another input box will appear after hitting the "Remove duplicates" button. There you can input the number of scrobbles to compare to.
This is described in the doc above: Using the script - step by step

Since June 2024, Last.fm has added rate-limiting. From my own testing I concluded that you can request about 20 pages per minute. Eventually you'll run into this:

Page not available
You’re requesting too many pages
You're requesting a lot of pages which may slow down the site for other users. Why not take a break and enjoy a cup of tea?

This messes up the automated script, because it triggers the rate-limiting very fast, and then the script will continuously skip pages until you cancel it.

Hey @RudeySH, thanks for your findings!

I can confirm the script's misbehavior. This issue only started occurring recently.
Unless you plan to monitor the entire process manually and stop/restart the script (which temporarily resets the problem), we need a fix to handle this.

Potential solutions:

1. Increase Delay Before Reloading:

   • A longer delay between requests could prevent triggering the rate/request limit.
   • This might reduce the chance of encountering the rate-limit page, but may not fully solve the issue.

2. Detect Text on the Page:

   • The current script relies on URL-based detection, which is not working in this case. It detects the rate-limit page as a normal library page.
   • I should implement a method to search for specific text on each reload and stop the script when it finds the rate-limit text (or something similar).

Also, I couldn’t find any clear documentation on LastFM’s rate-limiting rules, which makes this trickier.
If we had more information, it could help refine the script and avoid hitting the limit.

---

Something different I wanted to write down:
I've noticed an issue when deleting duplicates: sometimes the "405 - Method Not Allowed" page appears.
Currently, the method used to handle this is by navigating back to the previous page in the browser history.
However, this only works if the "405" page doesn't reappear consecutively, as it may cause the history to point to an incorrect page (e.g., a wrong library page).

To improve this, the method should be updated to store the active page's URL (or at least the page number) and return to this stored page whenever a "405" page is detected.
This would ensure that the script can always return to the last functioning page and continue its operation smoothly.

I'll work on fixing these issues over the following days and will update you via a comment to keep you informed.
Any further suggestions or ideas are welcome and greatly appreciated.

I'm also maintaining a Last.fm script and have dealt with this issue myself.

I think the best course of action is to make the script detect the rate-limiting page, and automatically pauses for 10/20/30/etc. seconds before automatically trying again.

I advice against detecting the text on the page. The text might differ based on the user language. Instead, I would simply test if the page contains a <table> (or if you want to be really specific, a table with the chartlist CSS class).

A Last.fm script I wrote stores the amount of seconds to wait in a variable. The variable starts at 10 seconds. So, my Last.fm script only waits 10 seconds the first time it gets rate-limited. If it can successfully resume (without immediately getting rate-limited again) after 10 seconds, my script will continue loading pages. The next time it gets rate limited, it will still only wait 10 seconds. However, as soon as it detects rate-limiting immediately after waiting 10 seconds, it increases the variable by 10 seconds. From that moment onwards, the script will wait 20 seconds every time it gets rate-limited. When the script detects rate-limiting immediately after waiting 20 seconds, the variable is increased to 30 seconds. It goes on like that until the variable reaches 60 seconds. This approach might sound convoluted but it allows my script to load pages relatively fast without knowing the exact details of Last.fm's undocumented rate-limiting.

Hi @RudeySH,

thank you so much for giving tips. I've decided to go with a 5 second delay and increase it by another 5 seconds with each consecutive rate-limit.

---

New Release: Version 1.5.0!
Changes:

• Syntax Adjustments: Improvements made for better readability and performance.
• Type Fixes: Many variables now have corrected types for better functionality.
• Functional Changes: Fixed the page reload function and removed clutter during execution.
• Error Handling Improvement: Enhanced the script to correctly navigate back in case of errors.
• Rate-Limiting Fix: The awaited solution for rate-limiting issues is now implemented.

If you encounter any issues or have suggestions for further improvements, just let me know!