Skip to content

Instantly share code, notes, and snippets.

@ishan-marikar
Last active July 27, 2018 18:58
Show Gist options
  • Save ishan-marikar/0496b1a423f61b211afed75dfac9f00a to your computer and use it in GitHub Desktop.
Save ishan-marikar/0496b1a423f61b211afed75dfac9f00a to your computer and use it in GitHub Desktop.
async *scraper() {
let noMorePages = false;
let currentPageURL = this._searchURL;
while (!noMorePages) {
try {
let listingsForCurrentPage = await this._getListingsFromPage(
currentPageURL
);
const isLimited =
this._maxNumberOfPages &&
this._maxNumberOfPages === this._currentPageNumber;
const isLastPage =
listingsForCurrentPage && !listingsForCurrentPage.hasNext;
if (isLimited || isLastPage) {
noMorePages = !noMorePages;
return listingsForCurrentPage.listings;
} else {
yield listingsForCurrentPage.listings;
currentPageURL = listingsForCurrentPage.hasNext;
}
} catch (exception) {
throw exception;
}
}
}
@EdwardDrapkin
Copy link

EdwardDrapkin commented Jul 27, 2018

class IncompleteImplementation {
	constructor(firstUrl) {
		this.resultListeners = [];
		this.isDone = false;
		this.next = firstUrl;
	}
	
	async fetchPages() {
		const page = await getPageSomehow(this.next);
		this.resultListeners.forEach(listener => listener(page));
		
		if(!page.hasNext) {
			this.isDone = true;
		} else {
			this.next = page.hasNext;
			return await this.fetchPages();
		}	
	}
}

const whatever = new IncompleteImplementation('www.whatever.com');
whatever.resultListeners.push((page) => console.log(page));
whatever.fetchPages().then(() => console.log('All pages fetched!'));

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment