Skip to content

Instantly share code, notes, and snippets.

@martinratinaud
Last active April 7, 2023 10:49
Show Gist options
  • Select an option

  • Save martinratinaud/2cb888204ae1a47fb32eaf68619b56bf to your computer and use it in GitHub Desktop.

Select an option

Save martinratinaud/2cb888204ae1a47fb32eaf68619b56bf to your computer and use it in GitHub Desktop.
Check backlink existence and rel after buying them
import axios, { AxiosRequestConfig } from 'axios';
import { JSDOM } from 'jsdom';
export default class Scraper {
public JSDOM = JSDOM;
constructor() {}
async getUrl(url: string, axiosConfig?: AxiosRequestConfig) {
const { headers, ...options } = axiosConfig || {};
const config: AxiosRequestConfig = {
method: 'GET',
headers: {
// https://oxylabs.io/blog/5-key-http-headers-for-web-scraping
'Accept-Encoding': '*',
Accept: 'test/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
Cookie: '',
...(headers || {}),
},
timeout: 5000,
...options,
};
const { data } = await axios(url, config);
return data;
}
async getUrlDom(url: string, axiosConfig?: AxiosRequestConfig, options?: GetUrlConfig) {
const data = await this.getUrl(url, axiosConfig, options);
return new JSDOM(data);
}
}
@martinratinaud
Copy link
Copy Markdown
Author

Here is what it will look like in the terminal
Screen Shot 2023-04-07 at 14 45 28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment