Python web scraper that gets the new / top 50 charts, also allows you to play tracks using the selenium browser

soundscraper.py

from selenium import webdriver
import requests
import bs4
import os

# new, top, or mix url
top_url = "https://soundcloud.com/charts/top"
new_url = "https://soundcloud.com/charts/new"
track_url = "https://soundcloud.com/search/sounds?q="
artist_url = "https://soundcloud.com/search/people?q="
mix_url_end = "&filter.duration=epic"

# create the selenium browser
browser = webdriver.Chrome('/Users/ian/Downloads/chromedriver')
browser.get("https://soundcloud.com")

# main menu
print()
print(">>> Welcome to the Python Soundcloud Scraper")
print(">>> Explore the Top / New & Hot Charts for all Genres")
print(">>> Search Soundcloud for Tracks, Artist, and Mixes")
print()

# new or top menu
while True:
    print(">>> Menu")
    print(">>> 1 - Search for a track")
    print(">>> 2 - Search for an artist")
    print(">>> 3 - Search for a mix")
    print(">>> 4 - Top charts")
    print(">>> 5 - New & hot charts")
    print(">>> 0 - Exit")
    print()
    choice = int(input(">>> Your choice: "))
    if choice == 0:
        browser.quit()
        break
    print()

    # search for a track
    if choice == 1:
        name = input("Name of the track: ")
        print()
        "%20".join(name.split(" "))
        browser.get(track_url + name)
        continue

    # search for an artist
    if choice == 2:
        name = input("Name of the artist: ")
        print()
        "%20".join(name.split(" "))
        browser.get(artist_url + name)
        continue

    if choice == 3:
        name = input("Name of the mix: ")
        print()
        "%20".join(name.split(" "))
        browser.get(track_url + name + mix_url_end)
        continue

    # genre menu
    while True:
        print(">>> Genres Available:")
        print()

        # genre menu
        url = ''
        if choice == 4: url = top_url
        else: url = new_url

        # parse the html with beautiful soup
        request = requests.get(url)
        soup = bs4.BeautifulSoup(request.text, "lxml")
        # print request.text

        genres = soup.select("a[href*=genre]")[2:]
        # print each genre

        genre_links = []

        # print out the available genres
        for index, genre in enumerate(genres):
            print(str(index) + ": " + genre.text)
            genre_links.append(genre.get("href"))

        print()
        choice = input(">>> Your choice (x to re-select chart type): ")
        print()

        if choice == 'x': break
        else: choice = int(choice)

        # print(genre_links[choice])

        url = "http://soundcloud.com" + genre_links[choice]
        request = requests.get(url)
        soup = bs4.BeautifulSoup(request.text, "lxml")

        tracks = soup.select("h2")[3:]
        track_links = []
        track_names = []
        # print(tracks)

        for index, track in enumerate(tracks):
            track_links.append(track.a.get("href"))
            track_names.append(track.text)
            print(str(index+1) + ": " + track.text)
            print()

        # song selection loop
        while True:
            choice = input(">>> Your choice (x to re-select genre): ")
            print()

            if choice == 'x': break
            else: choice = int(choice)-1

            print("Now playing: " + track_names[choice])
            print()

            browser.get("http://soundcloud.com" + track_links[choice])

print()
print("Goodbye!")
print()

Hey. I'm using Mac OS 10.15 with with Python ver. 3.7.4 It works fine when I run it on Jupyter notebook, but not on Sublime. Thanks for getting back, Mich

…

On Sun, Jan 26, 2020 at 7:03 PM Ian Annase ***@***.***> wrote: So, I'm wondering what I'm doing wrong. Nothing happens after I input my choice/option. I hit enter but then nothing (using sublime text). Thanks, I thought this was pretty interesting as I'm just getting started with coding in general. Which operating system and version of Python are you using? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://gist.github.com/8e801029bdd3e4e52837fe1f7996be31?email_source=notifications&email_token=AMMLO7SL3H6CSL4D3Y7MCIDQ7ZFIDA5CNFSM4KLV43P2YY3PNVWWK3TUL52HS4DFVNDWS43UINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAGAT22#gistcomment-3155885>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMMLO7WDTSVD4TIV3WOLZHLQ7ZFIDANCNFSM4KLV43PQ> .

Are you running the code inside of the Terminal by navigating to the directory the file is located in and using the command "python3 soundscraper.py"?

Yes, I am. I did forget to mention, the only thing I think I'm doing different is that I'm using firefox and then geckodriver.

…

On Mon, Jan 27, 2020 at 4:12 AM Ian Annase ***@***.***> wrote: Are you running the code inside of the Terminal by navigating to the directory the file is located in and using the command "python3 soundscraper.py"? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://gist.github.com/8e801029bdd3e4e52837fe1f7996be31?email_source=notifications&email_token=AMMLO7VKWBGBTSASG5ZHQ3DQ73FRDA5CNFSM4KLV43P2YY3PNVWWK3TUL52HS4DFVNDWS43UINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAGAU56#gistcomment-3156447>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMMLO7QGJJHGPIS6JQ2FFDTQ73FRDANCNFSM4KLV43PQ> .

Neve mind. It is working inside the terminal! thanks for your help. By the way, I've been inspired to do a project since working on this. I'll keep you posted, going to finish the second part of the tutorial. Thanks again

…

On Tue, Jan 28, 2020 at 2:57 AM Mich Iwata ***@***.***> wrote: Yes, I am. I did forget to mention, the only thing I think I'm doing different is that I'm using firefox and then geckodriver. On Mon, Jan 27, 2020 at 4:12 AM Ian Annase ***@***.***> wrote: > Are you running the code inside of the Terminal by navigating to the > directory the file is located in and using the command "python3 > soundscraper.py"? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://gist.github.com/8e801029bdd3e4e52837fe1f7996be31?email_source=notifications&email_token=AMMLO7VKWBGBTSASG5ZHQ3DQ73FRDA5CNFSM4KLV43P2YY3PNVWWK3TUL52HS4DFVNDWS43UINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAGAU56#gistcomment-3156447>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AMMLO7QGJJHGPIS6JQ2FFDTQ73FRDANCNFSM4KLV43PQ> > . >