thoop/.htaccess

iit2011081 · 2016-12-02T12:58:45Z

I am using zf2 and have apache installed on server. I have added the following lines in my .htaccess file.

RewriteEngine On

<IfModule mod_proxy_http.c>
    RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]
    RewriteCond %{QUERY_STRING} _escaped_fragment_
    
    # Only proxy the request to Prerender if it's a request for HTML
    RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/%{REQUEST_SCHEME}://%{HTTP_HOST}/$2 [P,L]
</IfModule>

But still it is not redirecting my url to corresponding prerender url. Even I have changed the User Agent to google bot and added the escaped_fragment as query paramter. Am I testing correctly or there is something wrong with this htaccess code ?

iit2011081 · 2016-12-05T06:41:31Z

How to pass username and password in htaccess if I am using basic auth

greenfm · 2016-12-11T00:07:32Z

Is there a way to have the page cached when a user visits a page? or should this be automatic and that I'm doing it wrong?

SherifElfadaly · 2017-03-05T15:39:03Z

It doesn't work when sharing in whatsapp even when adding the Whatsapp user agent.

<IfModule mod_headers.c>
    RequestHeader set X-Prerender-Token "%MYTOKEN%"
</IfModule>

<IfModule mod_rewrite.c>

  RewriteEngine On
    # If requested resource exists as a file or directory
      # (REQUEST_FILENAME is only relative in virtualhost context, so not usable)
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
      # Only exception is /index.htm, /index.html
        RewriteCond %{REQUEST_URI} !/index\.html?
      # Go to it as is
        RewriteRule ^ - [L]
    # If non existent
      # If path ends with / and is not just a single /, redirect to without the trailing /
        RewriteCond %{REQUEST_URI} ^.*/$
        RewriteCond %{REQUEST_URI} !^/$
        RewriteRule ^(.*)/$ $1 [R,QSA,L]
      # If path that is not empty or / or /index.htm or /index.html, redirect to /#!/path
        RewriteCond %{REQUEST_URI} !(/index\.html?|/|)$
        RewriteRule ^(.*)$ /#!$1 [R,QSA,NE,L]
      # If not /, redirect to it.
        RewriteCond %{REQUEST_URI} !^/$
        RewriteRule ^ / [R,QSA,L]

  # Handle Prerender.io
    RequestHeader set X-Prerender-Token "%MYTOKEN%"

    RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|Whatsapp|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
    RewriteCond %{QUERY_STRING} _escaped_fragment_
    RewriteCond %{QUERY_STRING} _escaped_fragment_=([^&]*)

  # Proxy the request
    RewriteRule ^ http://service.prerender.io/http://%{HTTP_HOST}%?_escaped_fragment_=%1 [P,L]
    
</IfModule>

denitto · 2017-03-27T17:35:07Z

interesting note I found re: iMessage User Agent. Hope this helps someone!

iMessage User Agent

thoop · 2017-06-16T15:53:02Z

I don't get notified when someone comments on this gist. Send us an email at [email protected] if anyone has any questions or issues.

thucnd · 2017-07-05T10:07:57Z

it doesn't work on my site

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
RewriteRule ^ - [L]

RequestHeader set X-Prerender-Token "mytoken"

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yandex(Bot|Images|Video|Media) [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L]

# Allow Robots.txt to pass through
RewriteRule ^robots.txt - [L]
RewriteRule ^ /index.html [L]

please help me to check config

sashatexb · 2018-01-06T12:19:38Z

If you have prerender working and running for all pages except the root(home) page.
Your .htaccess might have the URL set to /index.php or /index.shtml for the homepage. If so, you could change your rewrite rule to this:

RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(index\.php)?(.*) http://service.prerender.io/http://www.example.com$3 [P,L]

I added (index.php or index.shtml)? to capture that out if found and changed the $2 to $3. Don't forget to change the http://www.example.com to your domain.

atomalak · 2018-03-07T16:27:11Z

Hi i have used prerender for my angular project i did everything for activate prerender however not success

My web site url is:http://ugurdogrusoy.com/preparing/

firstly addred

Then added prerender token inside .htaccess you can see rule and token below

RewriteEngine On # If requested resource exists as a file or directory # (REQUEST_FILENAME is only relative in virtualhost context, so not usable) RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L]

# If non existent
# If path ends with / and is not just a single /, redirect to without the trailing /
RewriteCond %{REQUEST_URI} ^.*/$
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^(.*)/$ $1 [R,QSA,L]      

# Handle Prerender.io
RequestHeader set X-Prerender-Token "MY_TOKEN"
   

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

# Proxy the request
RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}/preparing/$1 [P,L]

# If non existent
# Accept everything on index.html
RewriteRule ^ /index.html

i can't escape We haven't seen a request with your prerender token yet

where have i done wrong you can help me ?

Thanks

Bersam · 2018-05-27T11:40:27Z

Any Idea how to integrate that with tomcat? seems P flag in Tomcat doesn't work.

mkosir · 2018-12-29T20:47:38Z

If anybody else is having a problem with always returning index file if it's not a crawler, so your SPA router can catch 404, I created a small gist -
https://gist.github.com/markokosir/77e8df3b7cf0b00023178c907f5c998e

d8bauxit8 · 2019-01-17T16:36:42Z

A warning to anyone using vector files. You will need to add a |\.svg capture group to the proxy rule.

BorisPocatko · 2019-01-30T02:44:56Z

If anyone needs to redirect .html files only (skipping extensions .xhtml,.shtml,.phtml,.jp,.cgi,.ppl,.pl,.perl,.php)
Here is my fork: https://gist.github.com/BorisPocatko/77f8f3812d05b934556854267a48150f

Volodymyrkohut · 2019-02-21T14:49:25Z

.htaccess

RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
# Go to it as is
RewriteRule ^ - [L]

  RewriteCond %{REQUEST_URI} ^.*/$
  RewriteCond %{REQUEST_URI} !^/$
  RewriteRule ^(.*)/$ $1 [R,QSA,L]      

RequestHeader set X-Prerender-Token "%MYTOKEN%"

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yandex(Bot|Images|Video|Media) [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L]

# Accept everything on index.html
RewriteRule ^ /index.html`

in index.html

But robots can't see my site.
On prerender.io i have 67 cached pages.

Could you please help me.

@Dizzyspb Did you fix it?

bobbymart1n · 2019-05-07T23:37:31Z

Hey all, I'm running into a problem when trying to Rewrite part of the URL being cached in prerender.io. I've tried removing that part of the URL using:
Rewrite ^root-directory/parent-directory/dist/(.*)$ $1 [R,L]
Doing this inside of:

<IfModule mod_proxy_http.c>
        Rewrite ^root-directory/parent-directory/dist/(.*)$ $1 [R,L]
        RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]
        RewriteCond %{QUERY_STRING} _escaped_fragment_

        # Only proxy the request to Prerender if it's a request for HTML
        RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/https://www.example.com/$2 [P,L]
    </IfModule>

Doing this on the live server results in TOO_MANY_REDIRECTS. Is there anyway to solve this? I've changed the https://www.example.com/$2 to represent my sites URL

eash-veera · 2019-07-07T08:57:58Z

Hi i have used prerender for my angular project i did everything for activate prerender however not success

My web site url is:http://ugurdogrusoy.com/preparing/

firstly addred

Then added prerender token inside .htaccess you can see rule and token below

RewriteEngine On # If requested resource exists as a file or directory # (REQUEST_FILENAME is only relative in virtualhost context, so not usable) RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L]
# If non existent
# If path ends with / and is not just a single /, redirect to without the trailing /
RewriteCond %{REQUEST_URI} ^.*/$
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^(.*)/$ $1 [R,QSA,L]      

# Handle Prerender.io
RequestHeader set X-Prerender-Token "MY_TOKEN"
   

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

# Proxy the request
RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}/preparing/$1 [P,L]

# If non existent
# Accept everything on index.html
RewriteRule ^ /index.html
i can't escape We haven't seen a request with your prerender token yet

where have i done wrong you can help me ?

Thanks

I have had the same issue, our site is using react, and we even tried to shift the config file further up to the root. Still the same issue. Any luck with this ?

yokoishioka · 2020-06-09T22:12:34Z

you are awesome!! i almost wept when i finally saw the token not used message go away. thanks!!

arnotixe · 2020-07-30T15:08:38Z

Just adding 2 cents:
On my hosting provider (Dreamhost), the [NC,OR] yielded strange results when just copy/pasting. On RewriteCond with only one line, I had to write it as
RewriteCond … googlebot|yandex|…|…|… [NC]
(re-writing the RewriteCond with one per line also worked, but not with [OR] on the first line. This would work:)

RewriteCond … googlebot [NC]
RewriteCond … yandex [NC,OR]
RewriteCond … WhatsApp [NC,OR]

note the seemingly missing OR on the first line

On the other hand, my second cent is that last WhatsApp entry - turns out WhatsApp does the scraping directly from inside the app (at least it did on my android phone today ;) So my full line is now
RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|WhatsApp [NC]

(And my full htaccess

    RewriteEngine On
        
        # https://stackoverflow.com/questions/18406156/redirect-all-to-index-php-using-htaccess
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_FILENAME} !-d
        RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|WhatsApp [NC]
        #        RewriteCond %{HTTP_USER_AGENT} facebookexternalhit|googlebot [NC]   MUST BE WRITTEN WITHOUT OR
        #        RewriteCond %{HTTP_USER_AGENT} googlebot [NC]
        #        RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC,OR]          'OR' IS FOR SECOND LINE (AND THIRD AND FOURTH ETC. WON'T WORK ON FIRST LINE)
        RewriteRule ^(.*)$ opengraph.php?q=$1 [NC,L,QSA]

        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_FILENAME} !-d
        RewriteRule ^(.*)$ redir.php?orig_path=$1 [NC,L,QSA]

</IfModule>

tyluudinh · 2021-10-19T15:10:56Z

Angular 7 + Apache2: work for me. Ping me if you have any issue

.htaccess: https://gist.github.com/tyluudinh/2149d6cc62219c141f619682b5378c47

.apache2.conf: https://gist.github.com/tyluudinh/bc257a1aa2a4b8826ff28916238cc27d

Kotstar1 · 2022-02-11T13:18:46Z

@tyluudinh This worked for me, thank you 👍

thoop/.htaccess

iit2011081 commented Dec 2, 2016

iit2011081 commented Dec 5, 2016

greenfm commented Dec 11, 2016

SherifElfadaly commented Mar 5, 2017

denitto commented Mar 27, 2017

thoop commented Jun 16, 2017

thucnd commented Jul 5, 2017 •

edited

Loading

sashatexb commented Jan 6, 2018 •

edited

Loading

atomalak commented Mar 7, 2018 •

edited

Loading

Bersam commented May 27, 2018

mkosir commented Dec 29, 2018 •

edited

Loading

d8bauxit8 commented Jan 17, 2019

BorisPocatko commented Jan 30, 2019

Volodymyrkohut commented Feb 21, 2019 •

edited

Loading

bobbymart1n commented May 7, 2019 •

edited

Loading

eash-veera commented Jul 7, 2019

yokoishioka commented Jun 9, 2020

arnotixe commented Jul 30, 2020

tyluudinh commented Oct 19, 2021

Kotstar1 commented Feb 11, 2022

	# Change YOUR_TOKEN to your prerender token
	# Change http://example.com (at the end of the last RewriteRule) to your website url

	<IfModule mod_headers.c>
	RequestHeader set X-Prerender-Token "YOUR_TOKEN"
	</IfModule>

	<IfModule mod_rewrite.c>
	RewriteEngine On

	<IfModule mod_proxy_http.c>
	RewriteCond %{HTTP_USER_AGENT} googlebot\|bingbot\|yandex\|baiduspider\|facebookexternalhit\|twitterbot\|rogerbot\|linkedinbot\|embedly\|quora\ link\ preview\|showyoubot\|outbrain\|pinterest\/0\.\|pinterestbot\|slackbot\|vkShare\|W3C_Validator [NC,OR]
	RewriteCond %{QUERY_STRING} _escaped_fragment_

	# Only proxy the request to Prerender if it's a request for HTML
	RewriteRule ^(?!.?(\.js\|\.css\|\.xml\|\.less\|\.png\|\.jpg\|\.jpeg\|\.gif\|\.pdf\|\.doc\|\.txt\|\.ico\|\.rss\|\.zip\|\.mp3\|\.rar\|\.exe\|\.wmv\|\.doc\|\.avi\|\.ppt\|\.mpg\|\.mpeg\|\.tif\|\.wav\|\.mov\|\.psd\|\.ai\|\.xls\|\.mp4\|\.m4a\|\.swf\|\.dat\|\.dmg\|\.iso\|\.flv\|\.m4v\|\.torrent\|\.ttf\|\.woff\|\.svg))(.) http://service.prerender.io/http://example.com/$2 [P,L]
	</IfModule>
	</IfModule>

thoop/.htaccess

iit2011081 commented Dec 2, 2016

iit2011081 commented Dec 5, 2016

greenfm commented Dec 11, 2016

SherifElfadaly commented Mar 5, 2017

denitto commented Mar 27, 2017

thoop commented Jun 16, 2017

thucnd commented Jul 5, 2017 • edited Loading

sashatexb commented Jan 6, 2018 • edited Loading

atomalak commented Mar 7, 2018 • edited Loading

Bersam commented May 27, 2018

mkosir commented Dec 29, 2018 • edited Loading

d8bauxit8 commented Jan 17, 2019

BorisPocatko commented Jan 30, 2019

Volodymyrkohut commented Feb 21, 2019 • edited Loading

bobbymart1n commented May 7, 2019 • edited Loading

eash-veera commented Jul 7, 2019

yokoishioka commented Jun 9, 2020

arnotixe commented Jul 30, 2020

tyluudinh commented Oct 19, 2021

Kotstar1 commented Feb 11, 2022

thucnd commented Jul 5, 2017 •

edited

Loading

sashatexb commented Jan 6, 2018 •

edited

Loading

atomalak commented Mar 7, 2018 •

edited

Loading

mkosir commented Dec 29, 2018 •

edited

Loading

Volodymyrkohut commented Feb 21, 2019 •

edited

Loading

bobbymart1n commented May 7, 2019 •

edited

Loading