ncouture

setup(
    entry_points={
        'paste.filter_app_factory': [
            'ssl_only = myapp.middlewares.ssl_only:make_filter',
        ],
    }
)

ncouture

# Example tidy-html5 configuration file.

fix-uri: yes
keep-time: yes
hide-comment: yes
indent-cdata: yes
omit-optional-tags: yes
output-encoding: utf8
indent: auto
indent-spaces: 2

ncouture

Info

This guide sets up a non-clustered Nutch crawler, which stores its data via HBase. We will not learn how to setup Hadoop et al., but just the bare minimum to crawl and index websites on a single machine.

Terms

• Nutch - the crawler (fetches and parses websites)
• HBase - filesystem storage for Nutch (Hadoop component, basically)