dtaivpp

Local Semantic Search in OpenSearch!

If you'd like to watch this demo through it's available on YouTube or if you prefer reading there is a walkthrough on my blog.

Cluster settings for Amazon OpenSearch and Locally running

PUT /_cluster/settings
{
    "persistent": {
        "plugins.ml_commons.only_run_on_ml_node": false,

dtaivpp

#!/bin/bash
# 1. To run this script curl it locally:
#    curl https://gist.githubusercontent.com/dtaivpp/c587d99a2cab441eba0314534ae87c86/raw -o opensearch-compose-bootstrap.sh
# 2. Change it to be executable:
#    chmod +x opensearch-compose-generate.sh
# 3. Run it:
#    ./opensearch-compose-generate.sh 
#
# This will create:
#     - docker-compose.yml file for OpenSearch 

dtaivpp

Semantic Search with OpenSearch and Cohere

Cluster Settings:

PUT /_cluster/settings
{
    "persistent": {
        "plugins.ml_commons.allow_registering_model_via_url": true,
        "plugins.ml_commons.only_run_on_ml_node": false,
        "plugins.ml_commons.connector_access_control_enabled": true,

dtaivpp

Find the image to save

docker images

Save to destination

docker save -o /path/to/output.tar org/container

Import

docker docker load -i /path/to/output.tar

dtaivpp

services:
  opensearch:
    image: opensearchproject/opensearch:${OPENSEARCH_VERSION:-2.11.1}
    container_name: opensearch
    environment:
      discovery.type: single-node
      node.name: opensearch
      OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
    volumes:
      - opensearch-data:/usr/share/opensearch/data

dtaivpp

# Download this file with:
# curl -L https://gist.githubusercontent.com/dtaivpp/a9b00957aa7d9cfe33e92aff8d50c835/raw/logging_demo.py -o logging_demo.py
import logging

#
# Named loggers
#
logger = logging.getLogger(__name__)
print(logger.name)

dtaivpp

version = 2
  [plugins]
    [plugins."io.containerd.grpc.v1.cri"]
      [plugins."io.containerd.grpc.v1.cri".containerd]
        default_runtime_name = "nvidia"

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
            privileged_without_host_devices = false
            runtime_engine = ""

dtaivpp

usage: ghdorker [-h] [-v] [-s {repo,user,org}] [-d DORKS] [--debug] [-o OUTPUT_FILENAME] [--options INPUT_OPTION [INPUT_OPTION ...]] search

Search github for github dorks

positional arguments:
  search                The repo, username, or organization you would like to search

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

dtaivpp

import yaml
import os

yaml_files = []

rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir):
  for fname in fileList:
    if fname[-4:] == 'yaml':
     yaml_files.append( dirName + "/" + fname)

dtaivpp

{
    "name": "Python 3",
    "build": {
        "dockerfile": "Dockerfile",
        "context": "..",
        "args": { 
            // Update 'VARIANT' to pick a Python version: 3, 3.6, 3.7, 3.8, 3.9
            "VARIANT": "3",
            // Options
            "INSTALL_NODE": "true",