qhuang872

/**
 * Global Reset of all HTML Elements
 *
 * Resetting all of our HTML Elements ensures a smoother
 * visual transition between browsers. If you don't believe me,
 * try temporarily commenting out this block of code, then go
 * and look at Mozilla versus Safari, both good browsers with
 * a good implementation of CSS. The thing is, all browser CSS
 * defaults are different and at the end of the day if visual
 * consistency is what we're shooting for, then we need to

qhuang872

Original error happens on line 11, that exit_ok was default to be False which caused the file_exist erorr, I made the change to set it to be True.
Also added a docstring for the download_file function.

qhuang872

Original error happens on line 11, that exit_ok was default to be False which caused the file_exist erorr, I made the change to set it to be True.
Also added a docstring for the download_file function.

qhuang872

import os
import urllib.request
from urllib.parse import urlparse


def download_file(url, cachedir=None, ignorecache=False, filename=None) -> str:
"""download_file would create a directory if not exist, then download the file to directory"""
    if not cachedir:
        cachedir = os.path.join(os.getcwd(), "build")

qhuang872

1, average runtime of the program is 52 seconds.

2, I separate the program into 4 different small functions, each serving only one purposes: read_content, get_link, download_video and download_image. So that I can measure memory usage of each part, using memory_profiler. According to the output of memory_profiler, read_content takes up the most memory: about 48 MiB. Other functions and their memory usage:  get_link uses 40 MiB, download_video uses 42 MiB, and download_image uses 45 MiB.

3, in this program, variable contents which is a json object occupies most memory.

4, 1) the first way to optimize this script is to skip the "get_link" process, which means instead of  storing all mp4_links and png_links into lists, since the "get_link" process is just a for loop, by eliminating this process, we are reducing runtime as well. We can just iterate through every json object, and just download the link from there. However, this solution only reduces runtime by 1~2 seconds in average, in my opinion, this is not