Skip to content

Instantly share code, notes, and snippets.

View manisnesan's full-sized avatar
🎯
Focusing

Manikandan Sivanesan` manisnesan

🎯
Focusing
View GitHub Profile
import requests
UNIFIED_FIELDS = ["title", "text"]
CASE_FIELDS = ["summary", "product", "description"]
SOLUTION_FIELDS = [
"title", "solution_environment", "issue",
"solution_rootcause", "solution_diagnosticsteps",
"solution_resolution"
]
@manisnesan
manisnesan / qna_as_pydantic.ipynb
Last active October 23, 2024 10:20
Qna.yaml as Pydantic Data Model with YAML to Pydantic Conversions This script defines a Pydantic data model for a Q&A system based on a YAML structure. It includes classes for handling questions and answers, document chunks, and overall document metadata. The model supports validation for SHA1 commit hashes.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@manisnesan
manisnesan / 50-rag-mvp.ipynb
Created August 24, 2024 20:28
Ben Clavie - Beyond RAG MVP
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@manisnesan
manisnesan / test_flashrank_bm25_scifact.ipynb
Created August 24, 2024 20:25
Evaluation notebook on scifact dataset using bm25 and flashrank along with examples of ranx and reranker library
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
# Install ujson
from fastcore.utils import dumps
def write_jsonl(cases: list, filename: str)->None:
''' Write a list of dictionaries to a jsonl file. This uses fastcore.utils `dumps` to serialize the dictionary to a string with `ujson` module.
Install `ujson` for faster serialization.'''
with open(filename, 'w') as file:
for item in cases: file.write(dumps(item) + '\n')
def read_jsonl(fname)->list:
''' Read a jsonl file into a list of dictionaries. This uses `ujson` module for faster deserialization. Install `ujson` for faster deserialization.'''
@manisnesan
manisnesan / 021_caikit_tutorial.ipynb
Created October 11, 2023 19:06
021_caikit_tutorial.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@manisnesan
manisnesan / pro-tips.md
Last active February 27, 2022 18:28
Pro Tips and Exercices

Week 1

  • Creating multiple fields is also helpful if you want to use a field for autocompletion, search-as-you-type, or joins.
  • If you only need a field for ranking, you might consider using the Rank Feature field and query for improved performance.
  • build a more sophisticated query by adding and grouping different types of queries via things like the “bool” query.
  • Mappings : Take an iterative approach to field mappings. That is, start by indexing the data using a subset of the content and the default settings. Then look to see what OpenSearch guessed for mappings and then modify those values accordingly to the requirements above and your insights.
  • Look for additional fields we could either search or leverage in our query. Eg: manufacturer, color that satisifes the user intent.
  • Using function query to impleme
@manisnesan
manisnesan / altering_results.dev
Created February 22, 2022 01:03
Altering results using scripting and function scores
## Altering resuts using painless - https://www.elastic.co/guide/en/elasticsearch/painless/7.10/painless-walkthrough.html
### Script score on price for doc 1
### In these results, doc_a gets a score of price (5.99) + 1 (due to the match_all score) = 6.99. All else gets a score of 1 (match_all score) + 1 (the non “doc_a” case in the script) = 2.
POST searchml_test/_search
{
"query": {"match_all": {}},
"rescore": {