Skip to content

Instantly share code, notes, and snippets.

@coolstar

coolstar/1. DepictionScrapers.plist format

Last active February 9, 2021 00:08
Show Gist options
  • Save coolstar/ffaf45f88354599a85d8a68a191fba7e to your computer and use it in GitHub Desktop.
Save coolstar/ffaf45f88354599a85d8a68a191fba7e to your computer and use it in GitHub Desktop.
Download ZIP
depiction-convert-docs.md
Raw
1. DepictionScrapers.plist format
Plist contains a dictionary
Key: Name of Javascript file (without js extension)
Value: Dictionary of requirements
Requirements Dictionary Keys:
* host: The host name for the depiction [optional]
* prefix: A prefix to check for in the path (after the host name) [optional]
* suffix: A suffix to check for in the path [optional]
If the requirements specified in this dictionary are satisfied by the web depiction URL, your Depiction JS is loaded in a VM
Raw
2. Sileo Javascript API
The Depiction Conversion script runs in a headless Javascript VM. However, several Sileo APIs are available:
Global Objects:
* SileoGen: The Sileo Native Depiction Generation APIs (Convenience functions to generate Sileo depictions or help scrape certain items like screenshots)
- See: SileoGen.js in the Sileo app bundle
* html: The html for the legacy depiction
* head: an instance of ScraperElement that represents the <head> object in the legacy depiction
* body: an instance of ScraperElement that represents the <body> object in the legacy depiction
Global APIs:
* print(string): Prints out the contents of string to os_log (view in Console.app)
* cleanHTML(string): Cleans out unwanted HTML elements or excess line breaks from string and returns a new string containing the sanitized HTML
* absoluteURL(string): Accepts relative or absolute URLs and returns an absolute URL that can be used to load images/etc.
* downloadPage(string [absolute URL], string [global variable name to set for head], string [global variable name to set for body):
Downloads the contents of the URL specified and parses the HTML. Sets global variables specified with instances of ScraperElement for the head and body tags of the specified URL
Note: This API may only be used a total of 10 times per depiction to preserve bandwidth
Your Script must return a JSON String of the generated native depiction (the easiest way to do this is to JSON.stringify an object representation the root view).
Raw
3. Reference SileoGen.js
/*
Sileo Depiction Generator Utilities
Copyright (c) 2019, CoolStar. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software must display the following acknowledgement:
This product includes software developed by the Sileo Team.
4. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY COOLSTAR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDER BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
var SileoGen = new Object();
SileoGen.version = "1.0";
SileoGen.screenshotSizes = [
{width: 320, height: 480, cornerRadius: 16}, //iPhone 4/4S
{width: 320, height: 568, cornerRadius: 16}, //iPhone 5/5S/SE
{width: 375, height: 667, cornerRadius: 16}, //iPhone 6/6S/7/8
{width: 414, height: 736, cornerRadius: 16}, //iPhone 6+/6S+/7+/8+
{width: 375, height: 812, cornerRadius: 25}, //iPhone X/XS
{width: 414, height: 896, cornerRadius: 25} //iPhone XR/XS Max
];
SileoGen.mostCommonSize = {
width: 414,
height: 736,
cornerRadius: 16
};
SileoGen.generateHeader = function(text){
return {
"class": "DepictionHeaderView",
"title": text
};
};
SileoGen.generateSubheader = function(text){
return {
"class": "DepictionSubheaderView",
"title": text
};
};
SileoGen.generateLabel = function(text, fontSize, fontWeight, textColor){
return {
"class": "DepictionLabelView",
"text": text,
"fontSize": fontSize,
"fontWeight": fontWeight,
"textColor": textColor
};
}
SileoGen.generateSeparator = function(){
return {
"class": "DepictionSeparatorView"
};
};
SileoGen.trimSeparator = function(array){
if (array[array.length - 1].class == "DepictionSeparatorView"){
array.pop();
}
};
SileoGen.generateTableText = function(title, text){
return {
"class": "DepictionTableTextView",
"text": text,
"title": title
};
};
SileoGen.generateTableButton = function(title, action){
return {
"class": "DepictionTableButtonView",
"title": title,
"action": absoluteURL(action)
};
};
SileoGen.generateStackView = function(){
return {
"class": "DepictionStackView",
"views": Array()
};
};
SileoGen.generateAutostackView = function(horizontalSpacing){
return {
"class": "DepictionAutoStackView",
"horizontalSpacing": horizontalSpacing,
"views": Array()
};
}
SileoGen.generateMarkdown = function(html){
return {
"class": "DepictionMarkdownView",
"markdown": html
};
};
SileoGen.generateImage = function(URL, width, height){
return {
"class": "DepictionImageView",
"URL": absoluteURL(URL),
"width": width,
"height": height,
"cornerRadius": 5
};
};
SileoGen.generateScreenshot = function(url, accessibilityText){
return {
"url": absoluteURL(url),
"accessibilityText": accessibilityText
};
};
SileoGen.generateScreenshots = function(width, height, cornerRadius){
return {
"class": "DepictionScreenshotsView",
"itemCornerRadius": cornerRadius,
"itemSize": "{" + width + ", " + height + "}",
"screenshots": Array()
};
};
Raw
4. Example BigBoss.js
/*
BigBoss HTML Depiction Scraper
Copyright (c) 2019, CoolStar. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software must display the following acknowledgement:
This product includes software developed by the Sileo Team.
4. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY COOLSTAR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDER BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
var parseTag = function(el, type){
let tag = el.tag();
if (tag == "label"){
if (el.text().includes("Description")){
return null;
}
let labelText = SileoGen.generateHeader(el.text());
return labelText;
} else if (tag == "table"){
let stackView = SileoGen.generateStackView();
let rows = el.getElementsWithTag("tr");
rows.forEach(function(row, idx){
let key = row.getElementsWithClassName("key")[0];
let detail = row.getElementsWithClassName("detail")[0];
if (key == null || detail == null){
return;
}
if (key.tag() != "td" || detail.tag() != "td"){
return;
}
let tableCell = SileoGen.generateTableText(key.text(), detail.text());
stackView.views.push(tableCell);
});
return stackView;
} else if (tag == "a" && el.className() == "button"){
let href = el.attr("href");
let button = SileoGen.generateTableButton(el.text(), href);
return button;
}
return null;
};
var parseMain = function(){
let panels = body.getElementsWithTag("panel");
let detailsStackView = SileoGen.generateStackView();
detailsStackView.tabname = "Details"
let changelogStackView = SileoGen.generateStackView();
changelogStackView.tabname = "Changelog"
panels.forEach(function(panel, idx){
let stackView = detailsStackView;
let fieldsets = panel.getElementsWithTag("fieldset");
fieldsets.forEach(function(fieldset, idx){
let tables = fieldset.getElementsWithTag("table");
let elements = fieldset.children();
elements.forEach(function(el, idx){
if (el.tag() == "div" && tables.length == 0) {
let autoStackView = SileoGen.generateAutostackView(10);
let screenshotsArr = new Array();
let lastSize = {width: NaN, height: NaN, cornerRadius: 0};
let images = el.getElementsWithTag("img");
images.forEach(function(img, idx){
let width = parseFloat(img.attr("width"));
let height = parseFloat(img.attr("height"));
let src = img.attr("src");
let alt = img.attr("alt");
let aspectRatio = height / width;
var isScreenshot = false;
SileoGen.screenshotSizes.forEach(function(size, idx){
if (isNaN(height) || isNaN(width)){
isScreenshot = true;
}
let screenshotAspectRatio = size.height / size.width;
if (aspectRatio > screenshotAspectRatio - 0.01 && aspectRatio < screenshotAspectRatio + 0.01){
isScreenshot = true;
}
});
if (isScreenshot){
let screenshot = SileoGen.generateScreenshot(src, alt);
screenshotsArr.push(screenshot);
if (!(isNaN(width) && !isNaN(height))){
lastSize.width = width;
lastSize.height = height;
}
if (isNaN(lastSize.width)){
lastSize.width = width;
}
if (isNaN(lastSize.height)){
lastSize.height = height;
}
} else {
let image = SileoGen.generateImage(src, width, height);
image.preferredWidth = width;
autoStackView.views.push(image);
}
});
if (autoStackView.views.length > 0) {
stackView.views.push(autoStackView);
}
let cleanedHTML = cleanHTML(el.html());
let markdown = SileoGen.generateMarkdown(cleanedHTML);
markdown.useRawFormat = true;
print("HTML: " + cleanedHTML);
stackView.views.push(markdown);
let commonSize = SileoGen.mostCommonSize;
if (isNaN(lastSize.width) && !isNaN(lastSize.height)){
lastSize.width = lastSize.height * (commonSize.width/commonSize.height);
} else if (isNaN(lastSize.height) && !isNaN(lastSize.width)){
lastSize.height = lastSize.width * (commonSize.height/commonSize.width);
}
if (screenshotsArr.length > 0){
stackView.views.push(SileoGen.generateSeparator());
let screenshots = SileoGen.generateScreenshots(lastSize.width, lastSize.height, 8);
screenshots.screenshots = screenshotsArr;
stackView.views.push(screenshots);
}
} else if (el.tag() == "div"){
let divElements = el.children();
divElements.forEach(function(el, idx){
let item = parseTag(el);
if (item != null){
stackView.views.push(item);
}
});
} else {
let item = parseTag(el);
if (item != null){
stackView.views.push(item);
}
}
});
});
stackView.views.push(SileoGen.generateSeparator());
});
//SileoGen.trimSeparator(detailsStackView.views);
let origButton = SileoGen.generateTableButton("Original Depiction", ".");
detailsStackView.views.push(origButton);
let disclaimer = SileoGen.generateMarkdown("<span style='font-size: 12pt;color: #666666;text-align: center;'>This depiction has been automatically generated. It may be missing information.</span>");
disclaimer.useRawFormat = true;
detailsStackView.views.push(disclaimer);
changelogStackView.views.push(SileoGen.generateMarkdown("Changelogs are not available for this package."));
let rootView = {
"class": "DepictionTabView",
"minVersion": "0.7",
"tabs": [detailsStackView, changelogStackView]
};
return JSON.stringify(rootView);
}
parseMain();
Raw
5. Example MacCiti.js
/*
MacCiti HTML Depiction Scraper
Copyright (c) 2019, CoolStar. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software must display the following acknowledgement:
This product includes software developed by the Sileo Team.
4. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY COOLSTAR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL COPYRIGHT HOLDER BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
var bannerURL = "";
var parseTag = function(el, changelogStackView){
let tag = el.tag();
if (tag == "label"){
if (el.text().includes("Description")){
return null;
}
let labelText = SileoGen.generateHeader(el.text());
return labelText;
} else if (tag == "a" && el.attr("target") == "_new"){
let href = el.attr("href");
if (el.text().includes("Screenshots")){
downloadPage(absoluteURL(href), "screenshotsHead", "screenshotsBody");
return parseScreenshots();
} else if (el.text().includes("Changes")){
downloadPage(absoluteURL(href), "changesHead", "changesBody");
parseChangelogs(changelogStackView);
return null;
}
let button = SileoGen.generateTableButton(el.text(), href);
return button;
} else if (tag == "img"){
let src = el.attr("src");
if (bannerURL == "") {
bannerURL = absoluteURL(src);
return null;
}
let image = SileoGen.generateImage(src, 738, 0);
return image;
}
return null;
};
var parseScreenshots = function(){
var screenshots = SileoGen.generateScreenshots(160, 284, 8);
let imgs = screenshotsBody.getElementsWithTag("img");
imgs.forEach(function(img, idx){
let src = img.attr("src");
var screenshot = SileoGen.generateScreenshot(src, "Screenshot");
screenshots.screenshots.push(screenshot);
});
return screenshots;
};
var parseChangelogs = function(changelogStackView){
var changes = changesBody.getElementsWithTag("div");
changes.forEach(function(change, idx){
let label = change.getElementsWithTag("label")[0];
if (label != null){
let title = "Version " + label.text().replace(":","");
let cleanedHTML = cleanHTML(change.html()).substr(label.text().length + 1);
var subHeaderView = SileoGen.generateSubheader(title);
subHeaderView.useBoldText = true;
subHeaderView.useBottomMargin = false;
changelogStackView.views.push(subHeaderView);
var markdownView = SileoGen.generateMarkdown(cleanedHTML);
markdownView.useSpacing = true;
changelogStackView.views.push(markdownView);
}
});
};
var parseMain = function(){
let panels = body.getElementsWithTag("panel");
let detailsStackView = SileoGen.generateStackView();
detailsStackView.tabname = "Details"
let changelogStackView = SileoGen.generateStackView();
changelogStackView.tabname = "Changelog"
panels.forEach(function(panel, idx){
let stackView = detailsStackView;
let fieldsets = panel.getElementsWithTag("fieldset");
fieldsets.forEach(function(fieldset, idx){
let elements = fieldset.children();
elements.forEach(function(el, idx){
if (el.tag() == "div" ) {
let cleanedHTML = cleanHTML(el.html());
let markdown = SileoGen.generateMarkdown(cleanedHTML);
markdown.useRawFormat = true;
stackView.views.push(markdown);
} else {
let item = parseTag(el, changelogStackView);
if (item != null){
stackView.views.push(item);
}
}
});
});
stackView.views.push(SileoGen.generateSeparator());
});
let origButton = SileoGen.generateTableButton("Original Depiction", ".");
detailsStackView.views.push(origButton);
let disclaimer = SileoGen.generateMarkdown("<span style='font-size: 12pt;color: #666666;text-align: center;'>This depiction has been automatically generated. It may be missing information.</span>");
disclaimer.useRawFormat = true;
detailsStackView.views.push(disclaimer);
changelogStackView.views.push(SileoGen.generateMarkdown("Changelogs are not available for this package."));
let rootView = {
"class": "DepictionTabView",
"minVersion": "0.7",
"headerImage": bannerURL,
"tabs": [detailsStackView, changelogStackView]
};
return JSON.stringify(rootView);
}
parseMain();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment