bacloud22

Build a simple search engine for a multilingual website

Websites with a lot of user generated content lately started to adopt and support multiple languages! I think this is great, because basically, Why not? Having this, some of them rely on "listings" textual content like: "title", "tags", "description" to index the whole content, and then offer an internal search engine for the follow visitors.

Among them, we have these websites that really the media content (images and videos) are the raw value delivered to the user (like Youtube). So a visitor looking for "A slim vs. a fat wrestling", will be interested in all media whatever their title, tags and description are, admitting they fall in his/her search. This case, particularly is on the rise and if you have such website and didn't start supporting multilingual search, then here is a starting point.

First we will be using NodeJS as a web server and MongoDB as a database. This is meant very basic just to illustrate how you would benefit from ou

bacloud22

// @flow

import { promisify } from 'util'
import { tidy } from 'htmltidy2'
import DOMPurify from 'isomorphic-dompurify'
import { createRequire } from 'module'
import { stringTransformer } from './strings.js'
const require = createRequire(import.meta.url)

const { LanguageDetectorBuilder } = require('../node_modules/@pemistahl/lingua/node/lingua')