Ben Clavie - Beyond RAG MVP

50-rag-mvp.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastcore.all import *\n",
    "from rich import print as rprint\n",
    "from rich import inspect as rinspect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# %pip install lancedb -qqq\n",
    "%pip install wikipedia-api -qqq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "From bclavie talk on Mastering LLM conference - [GitHub Gist](https://gist.githubusercontent.com/bclavie/f7b041328615d52cf5c0a9caaf03fd5e/raw/cee577612008fc5d94c1b36cc8436eaf1132be85/rag_mvp.py)\n",
    "\n",
    "```\n",
    "# Fetch some text content in two different categories\n",
    "from wikipediaapi import Wikipedia\n",
    "wiki = Wikipedia('RAGBot/0.0', 'en')\n",
    "docs = [{\"text\": x,\n",
    "         \"category\": \"person\"}\n",
    "        for x in wiki.page('Hayao_Miyazaki').text.split('\\n\\n')]\n",
    "docs += [{\"text\": x,\n",
    "         \"category\": \"film\"}\n",
    "         for x in wiki.page('Spirited_Away').text.split('\\n\\n')]\n",
    "\n",
    "# Enter LanceDB\n",
    "import lancedb\n",
    "from lancedb.pydantic import LanceModel, Vector\n",
    "from lancedb.embeddings import get_registry\n",
    "\n",
    "# Initialise the embedding model\n",
    "model_registry = get_registry().get(\"sentence-transformers\")\n",
    "model = model_registry.create(name=\"BAAI/bge-small-en-v1.5\")\n",
    "\n",
    "# Create a Model to store attributes for filtering\n",
    "class Document(LanceModel):\n",
    "    text: str = model.SourceField()\n",
    "    vector: Vector(384) = model.VectorField()\n",
    "    category: str\n",
    "\n",
    "db = lancedb.connect(\".my_db\")\n",
    "tbl = db.create_table(\"my_table\", schema=Document)\n",
    "\n",
    "# Embed the documents and store them in the database\n",
    "tbl.add(docs)\n",
    "\n",
    "# Generate the full-text (tf-idf) search index\n",
    "tbl.create_fts_index(\"text\")\n",
    "\n",
    "# Initialise a reranker -- here, Cohere's API one\n",
    "from lancedb.rerankers import CohereReranker\n",
    "\n",
    "reranker = CohereReranker()\n",
    "\n",
    "query = \"What is Chihiro's new name given to her by the witch?\"\n",
    "\n",
    "results = (tbl.search(query, query_type=\"hybrid\") # Hybrid means text + vector\n",
    ".where(\"category = 'film'\", prefilter=True) # Restrict to only docs in the 'film' category\n",
    ".limit(10) # Get 10 results from first-pass retrieval\n",
    ".rerank(reranker=reranker) # For the reranker to compute the final ranking\n",
    "          )\n",
    "\n",
    "results.to_pandas()\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "- wikiepediaapi\n",
    "- lancedb\n",
    "- sentence-transformers\n",
    "- pydantic\n",
    "- cohere reranker (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wikipediaapi import Wikipedia\n",
    "wiki = Wikipedia('RAGBOT/0.0', 'en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = wiki.page('Hayao_Miyazaki'); \n",
    "chunks = p.text.split('\\n\\n');\n",
    "docs = [ {'text': chunk, 'category': 'person'} for chunk in chunks ] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#19) [{'text': \"Hayao Miyazaki (宮崎 駿 or 宮﨑 駿, Miyazaki Hayao, Japanese: [mijaꜜzaki hajao]; born January 5, 1941) is a Japanese animator, filmmaker, and manga artist. A founder of Studio Ghibli, he has attained international acclaim as a masterful storyteller and creator of Japanese animated feature films, and is widely regarded as one of the most accomplished filmmakers in the history of animation.\\nBorn in Tokyo City in the Empire of Japan, Miyazaki expressed interest in manga and animation from an early age, and he joined Toei Animation in 1963. During his early years at Toei Animation he worked as an in-between artist and later collaborated with director Isao Takahata. Notable films to which Miyazaki contributed at Toei include Doggie March and Gulliver's Travels Beyond the Moon. He provided key animation to other films at Toei, such as Puss in Boots and Animal Treasure Island, before moving to A-Pro in 1971, where he co-directed Lupin the Third Part I alongside Takahata. After moving to Zuiyō Eizō (later known as Nippon Animation) in 1973, Miyazaki worked as an animator on World Masterpiece Theater, and directed the television series Future Boy Conan (1978). He joined Tokyo Movie Shinsha in 1979 to direct his first feature film The Castle of Cagliostro as well as the television series Sherlock Hound. In the same period, he began writing and illustrating the manga Nausicaä of the Valley of the Wind (1982–1994) and directed the 1984 film adaptation produced by Topcraft.\\nMiyazaki co-founded Studio Ghibli in 1985. He directed numerous films with Ghibli, including Laputa: Castle in the Sky (1986), My Neighbor Totoro (1988), Kiki's Delivery Service (1989), and Porco Rosso (1992). The films were met with critical and commercial success in Japan. Miyazaki's film Princess Mononoke was the first animated film ever to win the Japan Academy Film Prize for Picture of the Year, and briefly became the highest-grossing film in Japan following its release in 1997; its distribution to the Western world greatly increased Ghibli's popularity and influence outside Japan. His 2001 film Spirited Away became the highest-grossing film in Japanese history, winning the Academy Award for Best Animated Feature, and is frequently ranked among the greatest films of the 21st century. Miyazaki's later films—Howl's Moving Castle (2004), Ponyo (2008), and The Wind Rises (2013)—also enjoyed critical and commercial success. Following the release of The Wind Rises, Miyazaki announced his retirement from feature films, though he later returned to write and direct his twelfth feature film The Boy and the Heron (2023), for which he won his second Academy Award for Best Animated Feature.\\nMiyazaki's works are characterized by the recurrence of themes such as humanity's relationship with nature and technology, the wholesomeness of natural and traditional patterns of living, the importance of art and craftsmanship, and the difficulty of maintaining a pacifist ethic in a violent world. The protagonists of his films are often strong girls or young women, and several of his films present morally ambiguous antagonists with redeeming qualities. Miyazaki's works have been highly praised and awarded; he was named a Person of Cultural Merit for outstanding cultural contributions in November 2012, and received the Academy Honorary Award for his impact on animation and cinema in November 2014. Miyazaki has frequently been cited as an inspiration for numerous animators, directors, and writers.\", 'category': 'person'},{'text': 'Early life\\nHayao Miyazaki was born on January 5, 1941, in Tokyo City, Empire of Japan, the second of four sons. His father, Katsuji Miyazaki (born 1915), was the director of Miyazaki Airplane, his brother\\'s company, which manufactured rudders for fighter planes during World War II. The business allowed his family to remain affluent during Miyazaki\\'s early life. Miyazaki\\'s father enjoyed purchasing paintings and demonstrating them to guests, but otherwise had little known artistic understanding. He said that he was in the Imperial Japanese Army around 1940; after declaring to his commanding officer that he wished not to fight because of his wife and young child, he was discharged after a lecture about disloyalty. According to Miyazaki, his father often told him about his exploits, claiming that he continued to attend nightclubs after turning 70. Katsuji Miyazaki died on March 18, 1993. After his death, Miyazaki felt that he had often looked at his father negatively and that he had never said anything \"lofty or inspiring\". He regretted not having a serious discussion with his father, and felt that he had inherited his \"anarchistic feelings and his lack of concern about embracing contradictions\".', 'category': 'person'},{'text': 'Miyazaki has noted that some of his earliest memories are of \"bombed-out cities\". In 1944, when he was three years old, Miyazaki\\'s family evacuated to Utsunomiya. After the bombing of Utsunomiya in July 1945, he and his family evacuated to Kanuma. The bombing left a lasting impression on Miyazaki, then aged four. As a child, Miyazaki suffered from digestive problems, and was told that he would not live beyond 20, making him feel like an outcast. From 1947 to 1955, Miyazaki\\'s mother Yoshiko suffered from spinal tuberculosis; she spent the first few years in hospital before being nursed from home. Yoshiko was frugal, and described as a strict, intellectual woman who regularly questioned \"socially accepted norms\". She was closest with Miyazaki, and had a strong influence on him and his later work. Yoshiko Miyazaki died in July 1983 at the age of 72.\\nMiyazaki began school in 1947, at an elementary school in Utsunomiya, completing the first through third grades. After his family moved back to Suginami-ku, Miyazaki completed the fourth grade at Ōmiya Elementary School, and fifth grade at Eifuku Elementary School, which was newly established after splitting off from Ōmiya Elementary. After graduating from Eifuku as part of the first graduating class, he attended Ōmiya Junior High School. He aspired to become a manga artist, but discovered he could not draw people; instead, he only drew planes, tanks, and battleships for several years. Miyazaki was influenced by several manga artists, such as Tetsuji Fukushima, Soji Yamakawa and Osamu Tezuka. Miyazaki destroyed much of his early work, believing it was \"bad form\" to copy Tezuka\\'s style as it was hindering his own development as an artist. Around this time, Miyazaki would often see movies with his father, who was an avid moviegoer; memorable films for Miyazaki include Meshi (1951) and Tasogare Sakaba (1955).\\nAfter graduating from Ōmiya Junior High, Miyazaki attended Toyotama High School. During his third and final year, Miyazaki\\'s interest in animation was sparked by Panda and the Magic Serpent (1958), Japan\\'s first feature-length animated film in color; he had sneaked out to watch the film instead of studying for his entrance exams. Miyazaki later recounted that he fell in love with the film\\'s heroine, Bai-Niang, and that the film moved him to tears and left a profound impression; he wrote that he was \"moved to the depths of [his] soul\" and that the \"pure, earnest world of the film\" affirmed a side of him that \"yearned desperately to affirm the world rather than negate it\". After graduating from Toyotama, Miyazaki attended Gakushuin University in the department of political economy, majoring in Japanese Industrial Theory. He joined the \"Children\\'s Literature Research Club\", the \"closest thing back then to a comics club\"; he was sometimes the sole member of the club. In his free time, Miyazaki would visit his art teacher from middle school and sketch in his studio, where the two would drink and \"talk about politics, life, all sorts of things\". Around this time, he also drew manga; he never completed any stories, but accumulated thousands of pages of the beginnings of stories. He also frequently approached manga publishers to rent their stories. In 1960, Miyazaki was a bystander during the Anpo protests, having developed an interest after seeing photographs in Asahi Graph; by that point, he was too late to participate in the demonstrations. Miyazaki graduated from Gakushuin in 1963 with degrees in political science and economics.', 'category': 'person'},{'text': \"Career\\nEarly career\\nIn 1963, Miyazaki was employed at Toei Animation; this was the last year the company hired regularly. After gaining employment, he began renting a four-and-a-half tatami (7.4 m2; 80 sq ft) apartment in Nerima, Tokyo; the rent was ¥6,000. His salary at Toei was ¥19,500. Miyazaki worked as an in-between artist on the theatrical feature anime Doggie March and the television anime Wolf Boy Ken (both 1963). He also worked on Gulliver's Travels Beyond the Moon (1965). He was a leader in a labor dispute soon after his arrival, and became chief secretary of Toei's labor union in 1964. Miyazaki later worked as chief animator, concept artist, and scene designer on The Great Adventure of Horus, Prince of the Sun (1968). Throughout the film's production, Miyazaki worked closely with his mentor, Yasuo Ōtsuka, whose approach to animation profoundly influenced Miyazaki's work. Directed by Isao Takahata, with whom Miyazaki would continue to collaborate for the remainder of his career, the film was highly praised, and deemed a pivotal work in the evolution of animation. Miyazaki moved to a residence in Ōizumigakuenchō in April 1969, after the birth of his second son.\\nMiyazaki provided key animation for The Wonderful World of Puss 'n Boots (1969), directed by Kimio Yabuki. He created a 12-chapter manga series as a promotional tie-in for the film; the series ran in the Sunday edition of Tokyo Shimbun from January to March 1969. Miyazaki later proposed scenes in the screenplay for Flying Phantom Ship (1969), in which military tanks would cause mass hysteria in downtown Tokyo, and was hired to storyboard and animate the scenes. Under the pseudonym Akitsu Saburō (秋津 三朗), Miyazaki wrote and illustrated the manga People of the Desert, published in 26 installments between September 1969 and March 1970 in Boys and Girls Newspaper (少年少女新聞, Shōnen shōjo shinbun). He was influenced by illustrated stories such as Fukushima's Evil Lord of the Desert (沙漠の魔王, Sabaku no maō). In 1970, Miyazaki moved residence to Tokorozawa. In 1971, he developed structure, characters and designs for Hiroshi Ikeda's adaptation of Animal Treasure Island; he created the 13-part manga adaptation, printed in Tokyo Shimbun from January to March 1971. Miyazaki also provided key animation for Ali Baba and the Forty Thieves.\\nMiyazaki left Toei Animation in August 1971, and was hired at A-Pro, where he directed, or co-directed with Takahata, 23 episodes of Lupin the Third Part I, often using the pseudonym Teruki Tsutomu (照樹 務). The two also began pre-production on a series based on Astrid Lindgren's Pippi Longstocking books, designing extensive storyboards; the series was canceled after Miyazaki and Takahata were unable to meet with Lindgren, and permission was refused to complete the project. In 1972 and 1973, Miyazaki wrote, designed and animated two Panda! Go, Panda! shorts, directed by Takahata. After moving from A-Pro to Zuiyō Eizō in June 1973, Miyazaki and Takahata worked on World Masterpiece Theater, which featured their animation series Heidi, Girl of the Alps, an adaptation of Johanna Spyri's Heidi. Zuiyō Eizō continued as Nippon Animation in July 1975. Miyazaki also directed the television series Future Boy Conan (1978), an adaptation of Alexander Key's The Incredible Tide.\", 'category': 'person'},{'text': 'Breakthrough films\\nMiyazaki left Nippon Animation in 1979, during the production of Anne of Green Gables; he provided scene design and organization on the first fifteen episodes. He moved to Telecom Animation Film, a subsidiary of TMS Entertainment, to direct his first feature anime film, The Castle of Cagliostro (1979), a Lupin III film. In his role at Telecom, Miyazaki helped train the second wave of employees. Miyazaki directed six episodes of Sherlock Hound in 1981, until issues with Sir Arthur Conan Doyle\\'s estate led to a suspension in production; Miyazaki was busy with other projects by the time the issues were resolved, and the remaining episodes were directed by Kyosuke Mikuriya. They were broadcast from November 1984 to May 1985. Miyazaki also wrote the graphic novel The Journey of Shuna, inspired by the Tibetan folk tale \"Prince who became a dog\". The novel was published by Tokuma Shoten in June 1983, dramatized for radio broadcast in 1987, and published in English as Shuna\\'s Journey in 2022. Hayao Miyazaki\\'s Daydream Data Notes was also irregularly published from November 1984 to October 1994 in Model Graphix; selections of the stories received radio broadcast in 1995.\\nAfter the release of The Castle of Cagliostro, Miyazaki began working on his ideas for an animated film adaptation of Richard Corben\\'s comic book Rowlf and pitched the idea to Yutaka Fujioka at TMS. In November 1980, a proposal was drawn up to acquire the film rights. Around that time, Miyazaki was also approached for a series of magazine articles by the editorial staff of Animage. During subsequent conversations, he showed his sketchbooks and discussed basic outlines for envisioned animation projects with editors Toshio Suzuki and Osamu Kameyama, who saw the potential for collaboration on their development into animation. Two projects were proposed: Warring States Demon Castle (戦国魔城, Sengoku ma-jō), to be set in the Sengoku period; and the adaptation of Corben\\'s Rowlf. Both were rejected, as the company was unwilling to fund anime projects not based on existing manga, and the rights for the adaptation of Rowlf could not be secured. An agreement was reached that Miyazaki could start developing his sketches and ideas into a manga for the magazine with the proviso that it would never be made into a film. The manga—titled Nausicaä of the Valley of the Wind—ran from February 1982 to March 1994. The story, as re-printed in the tankōbon volumes, spans seven volumes for a combined total of 1060 pages. Miyazaki drew the episodes primarily in pencil, and it was printed monochrome in sepia-toned ink. Miyazaki resigned from Telecom Animation Film in November 1982.', 'category': 'person'},{'text': \"Following the completion of Nausicaä of the Valley of the Wind's first two volumes, Suzuki and the other editors of Animage encouraged Miyazaki to work on a film adaptation; some documentaries claim he began writing the manga after his film pitch was rejected, but Miyazaki said the manga came first. Miyazaki's imagination was sparked by the mercury poisoning of Minamata Bay and how nature responded and thrived in a poisoned environment, using it to create the film's polluted world. By this time, Miyazaki had moved to the animation studio Topcraft and was finding some of the staff to be unreliable. He eventually decided to bring on several of his previous collaborators for the film's production, including Takahata who would serve as producer. Pre-production began on May 31, 1983; Miyazaki encountered difficulties in creating the screenplay, with only sixteen chapters of the manga to work with. Takahata enlisted experimental and minimalist musician Joe Hisaishi to compose the film's score. Nausicaä of the Valley of the Wind was released on March 11, 1984. It grossed ¥1.48 billion at the box office, and made an additional ¥742 million in distribution income. It is often seen as Miyazaki's pivotal work, cementing his reputation as an animator. It was lauded for its positive portrayal of women, particularly that of main character Nausicaä. Several critics have labeled Nausicaä of the Valley of the Wind as possessing anti-war and feminist themes; Miyazaki argues otherwise, stating that he only wishes to entertain. The successful cooperation on the creation of the manga and the film laid the foundation for other collaborative projects. In April 1984, Miyazaki opened his own office in Suginami Ward, naming it Nibariki.\", 'category': 'person'},{'text': 'Studio Ghibli\\nEarly films (1985–1996)\\nOn June 15, 1985, Miyazaki and Takahata founded the animation production company Studio Ghibli as a subsidiary of Tokuma Shoten. Studio Ghibli\\'s first film was Laputa: Castle in the Sky (1986), directed by Miyazaki. Some of the architecture in the film was also inspired by a Welsh mining town; Miyazaki witnessed the mining strike upon his first visit to Wales in 1984 and admired the miners\\' dedication to their work and community. Laputa was released on August 2, 1986, by the Toei Company. It sold around 775,000 tickets; Miyazaki and Suzuki expressed their disappointment with the film\\'s box office figures. Miyazaki\\'s following film, My Neighbor Totoro, was released alongside Takahata\\'s Grave of the Fireflies in April 1988 to ensure Studio Ghibli\\'s financial status. My Neighbor Totoro features the theme of the relationship between the environment and humanity, showing that harmony is the result of respecting the environment. While the film received critical acclaim, it was commercially unsuccessful at the box office. However, merchandising was successful, and the film was labeled as a cult classic.\\nIn 1987, Studio Ghibli acquired the rights to create a film adaptation of Eiko Kadono\\'s novel Kiki\\'s Delivery Service. Miyazaki\\'s work on My Neighbor Totoro prevented him from directing the adaptation; Sunao Katabuchi was chosen as director, and Nobuyuki Isshiki was hired as script writer. Miyazaki\\'s dissatisfaction of Isshiki\\'s first draft led him to make changes to the project, ultimately taking the role of director. Kadono was unhappy with the differences between the book and the screenplay. Miyazaki and Suzuki visited Kadono and invited her to the studio; she allowed the project to continue. The film was originally intended to be a 60-minute special, but expanded into a feature film after Miyazaki completed the storyboards and screenplay. Kiki\\'s Delivery Service premiered on July 29, 1989. It earned ¥2.15 billion at the box office, and was the highest-grossing film in Japan in 1989.\\nFrom March to May 1989, Miyazaki\\'s manga Hikōtei Jidai was published in the magazine Model Graphix. Miyazaki began production on a 45-minute in-flight film for Japan Airlines based on the manga; Suzuki ultimately extended the film into the feature-length film, titled Porco Rosso, as expectations grew. The outbreak of the Yugoslav Wars in 1991 affected Miyazaki, prompting a more sombre tone for the film; Miyazaki would later refer to the film as \"foolish\", as its mature tones were unsuitable for children. The film featured anti-war themes, which Miyazaki would later revisit. The airline remained a major investor in the film, resulting in its initial premiere as an in-flight film, prior to its theatrical release on July 18, 1992. The film was commercially successful and remained one of the highest-grossing films in Japan for several years.\\nStudio Ghibli set up its headquarters in Koganei, Tokyo in August 1992. In November 1992, two television spots directed by Miyazaki were broadcast by Nippon Television Network (NTV): Sora Iro no Tane, a 90-second spot adapted from the illustrated story Sora Iro no Tane by Rieko Nakagawa and Yuriko Omura; and Nandarou, a series of five advertisements featuring an undefinable creature. Miyazaki designed the storyboards and wrote the screenplay for Whisper of the Heart (1995), directed by Yoshifumi Kondō.', 'category': 'person'},{'text': 'Global emergence (1997–2008)\\nMiyazaki began work on the initial storyboards for Princess Mononoke in August 1994, based on preliminary thoughts and sketches from the late 1970s. While experiencing writer\\'s block during production, Miyazaki accepted a request for the creation of On Your Mark, a music video for the song of the same name by Chage and Aska. In the production of the video, Miyazaki experimented with computer animation to supplement traditional animation. On Your Mark premiered as a short before Whisper of the Heart. Despite the video\\'s popularity, Suzuki said that it was not given \"100 percent\" focus.', 'category': 'person'},{'text': 'In May 1995, Miyazaki took a group of artists and animators to the ancient forests of Yakushima and the mountains of Shirakami-Sanchi, taking photographs and making sketches. The landscapes in the film were inspired by Yakushima. In Princess Mononoke, Miyazaki revisited the ecological and political themes of Nausicaä of the Valley of the Wind. Miyazaki supervised the 144,000 cels in the film, about 80,000 of which were key animation. Princess Mononoke was produced with an estimated budget of ¥2.35 billion (approximately US$23.5 million), making it the most expensive Japanese animated film at the time. Approximately fifteen minutes of the film uses computer animation: about five minutes uses techniques such as 3D rendering, digital composition, and texture mapping; the remaining ten minutes uses digital ink and paint. While the original intention was to digitally paint 5,000 of the film\\'s frames, time constraints doubled this, though it remained below ten percent of the final film.\\nUpon its premiere on July 12, 1997, Princess Mononoke was critically acclaimed, becoming the first animated film to win the Japan Academy Film Prize for Picture of the Year. The film was also commercially successful, becoming the highest-grossing film in Japan for several months. Miramax Films purchased the film\\'s distributions rights for North America; while it was largely unsuccessful at the box office, grossing about US$2.3 million, it was seen as the introduction of Studio Ghibli to global markets. Miyazaki claimed Princess Mononoke would be his final film. Tokuma Shoten merged with Studio Ghibli in June 1997. Miyazaki left Studio Ghibli on January 14, 1998, to create a new studio called Butaya, to be succeeded by Kondō; however, Kondō\\'s death impacted Miyazaki, and he returned to Studio Ghibli on January 16, 1999.\\nMiyazaki\\'s next film was conceived while on vacation at a mountain cabin with his family and five young girls who were family friends. Miyazaki realized that he had not created a film for 10-year-old girls, and set out to do so. He read shōjō manga magazines like Nakayoshi and Ribon for inspiration, but felt they only offered subjects on \"crushes and romance\", which is not what the girls \"held dear in their hearts\". He decided to produce the film about a female heroine whom they could look up to. Production of the film, titled Spirited Away, commenced in 2000 on a budget of ¥1.9 billion (US$15 million). As with Princess Mononoke, the staff experimented with computer animation, but kept the technology at a level to enhance the story, not to \"steal the show\". Spirited Away deals with symbols of human greed, symbolizing the 1980s Japanese asset price bubble, and a liminal journey through the realm of spirits. The film was released on July 20, 2001; it received critical acclaim, and is considered among the greatest films of the 2000s. It won the Japan Academy Film Prize for Picture of the Year, and the Academy Award for Best Animated Feature. The film was also commercially successful, earning ¥30.4 billion (US$289.1 million) at the box office. It became the highest-grossing film in Japan, a record it maintained for almost 20 years. Following the death of Tokuma in September 2000, Miyazaki served as the head of his funeral committee.\\nIn September 2001, Studio Ghibli announced the production of Howl\\'s Moving Castle, based on the novel by Diana Wynne Jones. Mamoru Hosoda of Toei Animation was originally selected to direct the film, but disagreements between Hosoda and Studio Ghibli executives led to the project\\'s abandonment. After six months, Studio Ghibli resurrected the project. Miyazaki was inspired to direct the film upon reading Jones\\' novel, and was struck by the image of a castle moving around the countryside; the novel does not explain how the castle moved, which led to Miyazaki\\'s designs. He traveled to Colmar and Riquewihr in Alsace, France, to study the architecture and the surroundings for the film\\'s setting. Additional inspiration came from the concepts of future technology in Albert Robida\\'s work. It was released on November 20, 2004, and received widespread critical acclaim. The film received the Osella Award for Technical Excellence at the 61st Venice International Film Festival, and was nominated for the Academy Award for Best Animated Feature. In Japan, the film grossed a record $14.5 million in its first week of release. It remains among the highest-grossing films in Japan, with a worldwide gross of over ¥19.3 billion. Miyazaki received the honorary Golden Lion for Lifetime Achievement award at the 62nd Venice International Film Festival in 2005.\\nIn March 2005, Studio Ghibli split from Tokuma Shoten. In the 1980s, Miyazaki had contacted Ursula K. Le Guin expressing interest in producing an adaptation of her Earthsea novels; unaware of Miyazaki\\'s work, Le Guin declined. Upon watching My Neighbor Totoro several years later, Le Guin expressed approval to the concept of the adaptation. She met with Suzuki in August 2005, who wanted Miyazaki\\'s son Goro to direct the film, as Miyazaki had wished to retire. Disappointed that Miyazaki was not directing, but under the impression that he would supervise his son\\'s work, Le Guin approved of the film\\'s production. Miyazaki later publicly opposed and criticized Gorō\\'s appointment as director. Upon Miyazaki\\'s viewing of the film, he wrote a message for his son: \"It was made honestly, so it was good\".\\nMiyazaki designed the covers for several manga novels in 2006, including A Trip to Tynemouth; he also worked as editor, and created a short manga for the book. Miyazaki\\'s next film, Ponyo, began production in May 2006. It was initially inspired by \"The Little Mermaid\" by Hans Christian Andersen, though began to take its own form as production continued. Miyazaki aimed for the film to celebrate the innocence and cheerfulness of a child\\'s universe. He intended for it to only use traditional animation, and was intimately involved with the artwork. He preferred to draw the sea and waves himself, as he enjoyed experimenting. Ponyo features 170,000 frames—a record for Miyazaki. The film\\'s seaside village was inspired by Tomonoura, a town in Setonaikai National Park, where Miyazaki stayed in 2005. The main character, Sōsuke, is based on Gorō. Following its release on July 19, 2008, Ponyo was critically acclaimed, receiving Animation of the Year at the 32nd Japan Academy Film Prize. The film was also a commercial success, earning ¥10 billion (US$93.2 million) in its first month and ¥15.5 billion by the end of 2008, placing it among the highest-grossing films in Japan.', 'category': 'person'},{'text': 'Later films (2009–present)\\nIn early 2009, Miyazaki began writing a manga called Kaze Tachinu (風立ちぬ, The Wind Rises), telling the story of Mitsubishi A6M Zero fighter designer Jiro Horikoshi. The manga was first published in two issues of the Model Graphix magazine, published on February 25 and March 25, 2009. Miyazaki later co-wrote the screenplay for Arrietty (2010) and From Up on Poppy Hill (2011), directed by Hiromasa Yonebayashi and Gorō Miyazaki respectively. Miyazaki wanted his next film to be a sequel to Ponyo, but Suzuki convinced him to instead adapt Kaze Tachinu to film. In November 2012, Studio Ghibli announced the production of The Wind Rises, based on Kaze Tachinu, to be released alongside Takahata\\'s The Tale of the Princess Kaguya.\\nMiyazaki was inspired to create The Wind Rises after reading a quote from Horikoshi: \"All I wanted to do was to make something beautiful\". Several scenes in The Wind Rises were inspired by Tatsuo Hori\\'s novel The Wind Has Risen (風立ちぬ), in which Hori wrote about his life experiences with his fiancée before she died from tuberculosis. The female lead character\\'s name, Naoko Satomi, was borrowed from Hori\\'s novel Naoko (菜穂子). The Wind Rises continues to reflect Miyazaki\\'s pacifist stance, continuing the themes of his earlier works, despite stating that condemning war was not the intention of the film. The film premiered on July 20, 2013, and received critical acclaim; it was named Animation of the Year at the 37th Japan Academy Film Prize, and was nominated for Best Animated Feature at the 86th Academy Awards. It was also commercially successful, grossing ¥11.6 billion (US$110 million) at the Japanese box office, becoming the highest-grossing film in Japan in 2013.\\nIn September 2013, Miyazaki announced that he was retiring from the production of feature films due to his age, but wished to continue working on the displays at the Studio Ghibli Museum. Miyazaki was awarded the Academy Honorary Award at the Governors Awards in November 2014. He developed Boro the Caterpillar, an animated short film which was first discussed during pre-production for Princess Mononoke. It was screened exclusively at the Studio Ghibli Museum in July 2017. Around this time, Miyazaki was working on a manga titled Teppo Samurai. In February 2019, a four-part documentary was broadcast on the NHK network titled 10 Years with Hayao Miyazaki, documenting production of his films in his private studio. In 2019, Miyazaki approved a musical adaptation of Nausicaä of the Valley of the Wind, as it was performed by a kabuki troupe.\\nIn August 2016, Miyazaki proposed a new feature-length film, Kimi-tachi wa Dō Ikiru ka (titled The Boy and the Heron in English), on which he began animation work without receiving official approval. The film opened in Japanese theaters on July 14, 2023. It was preceded by a minimal marketing campaign, forgoing trailers, commercials, and advertisements, a response from Suzuki to his perceived oversaturation of marketing materials in mainstream films. Despite claims that The Boy and the Heron would be Miyazaki\\'s final film, Studio Ghibli vice president Junichi Nishioka said in September 2023 that Miyazaki continued to attend the office daily to plan his next film. Suzuki said he could no longer convince Miyazaki to retire. The Boy and the Heron won Miyazaki his second Academy Award for Best Animated Feature at the 96th Academy Awards, becoming the oldest director to do so; Miyazaki did not attend the show due to his advanced age.', 'category': 'person'}...]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs += [dict(text=chunk, category='film') for chunk in wiki.page('Spirited_Away').text.split('\\n\\n')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#46) [{'text': \"Hayao Miyazaki (宮崎 駿 or 宮﨑 駿, Miyazaki Hayao, Japanese: [mijaꜜzaki hajao]; born January 5, 1941) is a Japanese animator, filmmaker, and manga artist. A founder of Studio Ghibli, he has attained international acclaim as a masterful storyteller and creator of Japanese animated feature films, and is widely regarded as one of the most accomplished filmmakers in the history of animation.\\nBorn in Tokyo City in the Empire of Japan, Miyazaki expressed interest in manga and animation from an early age, and he joined Toei Animation in 1963. During his early years at Toei Animation he worked as an in-between artist and later collaborated with director Isao Takahata. Notable films to which Miyazaki contributed at Toei include Doggie March and Gulliver's Travels Beyond the Moon. He provided key animation to other films at Toei, such as Puss in Boots and Animal Treasure Island, before moving to A-Pro in 1971, where he co-directed Lupin the Third Part I alongside Takahata. After moving to Zuiyō Eizō (later known as Nippon Animation) in 1973, Miyazaki worked as an animator on World Masterpiece Theater, and directed the television series Future Boy Conan (1978). He joined Tokyo Movie Shinsha in 1979 to direct his first feature film The Castle of Cagliostro as well as the television series Sherlock Hound. In the same period, he began writing and illustrating the manga Nausicaä of the Valley of the Wind (1982–1994) and directed the 1984 film adaptation produced by Topcraft.\\nMiyazaki co-founded Studio Ghibli in 1985. He directed numerous films with Ghibli, including Laputa: Castle in the Sky (1986), My Neighbor Totoro (1988), Kiki's Delivery Service (1989), and Porco Rosso (1992). The films were met with critical and commercial success in Japan. Miyazaki's film Princess Mononoke was the first animated film ever to win the Japan Academy Film Prize for Picture of the Year, and briefly became the highest-grossing film in Japan following its release in 1997; its distribution to the Western world greatly increased Ghibli's popularity and influence outside Japan. His 2001 film Spirited Away became the highest-grossing film in Japanese history, winning the Academy Award for Best Animated Feature, and is frequently ranked among the greatest films of the 21st century. Miyazaki's later films—Howl's Moving Castle (2004), Ponyo (2008), and The Wind Rises (2013)—also enjoyed critical and commercial success. Following the release of The Wind Rises, Miyazaki announced his retirement from feature films, though he later returned to write and direct his twelfth feature film The Boy and the Heron (2023), for which he won his second Academy Award for Best Animated Feature.\\nMiyazaki's works are characterized by the recurrence of themes such as humanity's relationship with nature and technology, the wholesomeness of natural and traditional patterns of living, the importance of art and craftsmanship, and the difficulty of maintaining a pacifist ethic in a violent world. The protagonists of his films are often strong girls or young women, and several of his films present morally ambiguous antagonists with redeeming qualities. Miyazaki's works have been highly praised and awarded; he was named a Person of Cultural Merit for outstanding cultural contributions in November 2012, and received the Academy Honorary Award for his impact on animation and cinema in November 2014. Miyazaki has frequently been cited as an inspiration for numerous animators, directors, and writers.\", 'category': 'person'},{'text': 'Early life\\nHayao Miyazaki was born on January 5, 1941, in Tokyo City, Empire of Japan, the second of four sons. His father, Katsuji Miyazaki (born 1915), was the director of Miyazaki Airplane, his brother\\'s company, which manufactured rudders for fighter planes during World War II. The business allowed his family to remain affluent during Miyazaki\\'s early life. Miyazaki\\'s father enjoyed purchasing paintings and demonstrating them to guests, but otherwise had little known artistic understanding. He said that he was in the Imperial Japanese Army around 1940; after declaring to his commanding officer that he wished not to fight because of his wife and young child, he was discharged after a lecture about disloyalty. According to Miyazaki, his father often told him about his exploits, claiming that he continued to attend nightclubs after turning 70. Katsuji Miyazaki died on March 18, 1993. After his death, Miyazaki felt that he had often looked at his father negatively and that he had never said anything \"lofty or inspiring\". He regretted not having a serious discussion with his father, and felt that he had inherited his \"anarchistic feelings and his lack of concern about embracing contradictions\".', 'category': 'person'},{'text': 'Miyazaki has noted that some of his earliest memories are of \"bombed-out cities\". In 1944, when he was three years old, Miyazaki\\'s family evacuated to Utsunomiya. After the bombing of Utsunomiya in July 1945, he and his family evacuated to Kanuma. The bombing left a lasting impression on Miyazaki, then aged four. As a child, Miyazaki suffered from digestive problems, and was told that he would not live beyond 20, making him feel like an outcast. From 1947 to 1955, Miyazaki\\'s mother Yoshiko suffered from spinal tuberculosis; she spent the first few years in hospital before being nursed from home. Yoshiko was frugal, and described as a strict, intellectual woman who regularly questioned \"socially accepted norms\". She was closest with Miyazaki, and had a strong influence on him and his later work. Yoshiko Miyazaki died in July 1983 at the age of 72.\\nMiyazaki began school in 1947, at an elementary school in Utsunomiya, completing the first through third grades. After his family moved back to Suginami-ku, Miyazaki completed the fourth grade at Ōmiya Elementary School, and fifth grade at Eifuku Elementary School, which was newly established after splitting off from Ōmiya Elementary. After graduating from Eifuku as part of the first graduating class, he attended Ōmiya Junior High School. He aspired to become a manga artist, but discovered he could not draw people; instead, he only drew planes, tanks, and battleships for several years. Miyazaki was influenced by several manga artists, such as Tetsuji Fukushima, Soji Yamakawa and Osamu Tezuka. Miyazaki destroyed much of his early work, believing it was \"bad form\" to copy Tezuka\\'s style as it was hindering his own development as an artist. Around this time, Miyazaki would often see movies with his father, who was an avid moviegoer; memorable films for Miyazaki include Meshi (1951) and Tasogare Sakaba (1955).\\nAfter graduating from Ōmiya Junior High, Miyazaki attended Toyotama High School. During his third and final year, Miyazaki\\'s interest in animation was sparked by Panda and the Magic Serpent (1958), Japan\\'s first feature-length animated film in color; he had sneaked out to watch the film instead of studying for his entrance exams. Miyazaki later recounted that he fell in love with the film\\'s heroine, Bai-Niang, and that the film moved him to tears and left a profound impression; he wrote that he was \"moved to the depths of [his] soul\" and that the \"pure, earnest world of the film\" affirmed a side of him that \"yearned desperately to affirm the world rather than negate it\". After graduating from Toyotama, Miyazaki attended Gakushuin University in the department of political economy, majoring in Japanese Industrial Theory. He joined the \"Children\\'s Literature Research Club\", the \"closest thing back then to a comics club\"; he was sometimes the sole member of the club. In his free time, Miyazaki would visit his art teacher from middle school and sketch in his studio, where the two would drink and \"talk about politics, life, all sorts of things\". Around this time, he also drew manga; he never completed any stories, but accumulated thousands of pages of the beginnings of stories. He also frequently approached manga publishers to rent their stories. In 1960, Miyazaki was a bystander during the Anpo protests, having developed an interest after seeing photographs in Asahi Graph; by that point, he was too late to participate in the demonstrations. Miyazaki graduated from Gakushuin in 1963 with degrees in political science and economics.', 'category': 'person'},{'text': \"Career\\nEarly career\\nIn 1963, Miyazaki was employed at Toei Animation; this was the last year the company hired regularly. After gaining employment, he began renting a four-and-a-half tatami (7.4 m2; 80 sq ft) apartment in Nerima, Tokyo; the rent was ¥6,000. His salary at Toei was ¥19,500. Miyazaki worked as an in-between artist on the theatrical feature anime Doggie March and the television anime Wolf Boy Ken (both 1963). He also worked on Gulliver's Travels Beyond the Moon (1965). He was a leader in a labor dispute soon after his arrival, and became chief secretary of Toei's labor union in 1964. Miyazaki later worked as chief animator, concept artist, and scene designer on The Great Adventure of Horus, Prince of the Sun (1968). Throughout the film's production, Miyazaki worked closely with his mentor, Yasuo Ōtsuka, whose approach to animation profoundly influenced Miyazaki's work. Directed by Isao Takahata, with whom Miyazaki would continue to collaborate for the remainder of his career, the film was highly praised, and deemed a pivotal work in the evolution of animation. Miyazaki moved to a residence in Ōizumigakuenchō in April 1969, after the birth of his second son.\\nMiyazaki provided key animation for The Wonderful World of Puss 'n Boots (1969), directed by Kimio Yabuki. He created a 12-chapter manga series as a promotional tie-in for the film; the series ran in the Sunday edition of Tokyo Shimbun from January to March 1969. Miyazaki later proposed scenes in the screenplay for Flying Phantom Ship (1969), in which military tanks would cause mass hysteria in downtown Tokyo, and was hired to storyboard and animate the scenes. Under the pseudonym Akitsu Saburō (秋津 三朗), Miyazaki wrote and illustrated the manga People of the Desert, published in 26 installments between September 1969 and March 1970 in Boys and Girls Newspaper (少年少女新聞, Shōnen shōjo shinbun). He was influenced by illustrated stories such as Fukushima's Evil Lord of the Desert (沙漠の魔王, Sabaku no maō). In 1970, Miyazaki moved residence to Tokorozawa. In 1971, he developed structure, characters and designs for Hiroshi Ikeda's adaptation of Animal Treasure Island; he created the 13-part manga adaptation, printed in Tokyo Shimbun from January to March 1971. Miyazaki also provided key animation for Ali Baba and the Forty Thieves.\\nMiyazaki left Toei Animation in August 1971, and was hired at A-Pro, where he directed, or co-directed with Takahata, 23 episodes of Lupin the Third Part I, often using the pseudonym Teruki Tsutomu (照樹 務). The two also began pre-production on a series based on Astrid Lindgren's Pippi Longstocking books, designing extensive storyboards; the series was canceled after Miyazaki and Takahata were unable to meet with Lindgren, and permission was refused to complete the project. In 1972 and 1973, Miyazaki wrote, designed and animated two Panda! Go, Panda! shorts, directed by Takahata. After moving from A-Pro to Zuiyō Eizō in June 1973, Miyazaki and Takahata worked on World Masterpiece Theater, which featured their animation series Heidi, Girl of the Alps, an adaptation of Johanna Spyri's Heidi. Zuiyō Eizō continued as Nippon Animation in July 1975. Miyazaki also directed the television series Future Boy Conan (1978), an adaptation of Alexander Key's The Incredible Tide.\", 'category': 'person'},{'text': 'Breakthrough films\\nMiyazaki left Nippon Animation in 1979, during the production of Anne of Green Gables; he provided scene design and organization on the first fifteen episodes. He moved to Telecom Animation Film, a subsidiary of TMS Entertainment, to direct his first feature anime film, The Castle of Cagliostro (1979), a Lupin III film. In his role at Telecom, Miyazaki helped train the second wave of employees. Miyazaki directed six episodes of Sherlock Hound in 1981, until issues with Sir Arthur Conan Doyle\\'s estate led to a suspension in production; Miyazaki was busy with other projects by the time the issues were resolved, and the remaining episodes were directed by Kyosuke Mikuriya. They were broadcast from November 1984 to May 1985. Miyazaki also wrote the graphic novel The Journey of Shuna, inspired by the Tibetan folk tale \"Prince who became a dog\". The novel was published by Tokuma Shoten in June 1983, dramatized for radio broadcast in 1987, and published in English as Shuna\\'s Journey in 2022. Hayao Miyazaki\\'s Daydream Data Notes was also irregularly published from November 1984 to October 1994 in Model Graphix; selections of the stories received radio broadcast in 1995.\\nAfter the release of The Castle of Cagliostro, Miyazaki began working on his ideas for an animated film adaptation of Richard Corben\\'s comic book Rowlf and pitched the idea to Yutaka Fujioka at TMS. In November 1980, a proposal was drawn up to acquire the film rights. Around that time, Miyazaki was also approached for a series of magazine articles by the editorial staff of Animage. During subsequent conversations, he showed his sketchbooks and discussed basic outlines for envisioned animation projects with editors Toshio Suzuki and Osamu Kameyama, who saw the potential for collaboration on their development into animation. Two projects were proposed: Warring States Demon Castle (戦国魔城, Sengoku ma-jō), to be set in the Sengoku period; and the adaptation of Corben\\'s Rowlf. Both were rejected, as the company was unwilling to fund anime projects not based on existing manga, and the rights for the adaptation of Rowlf could not be secured. An agreement was reached that Miyazaki could start developing his sketches and ideas into a manga for the magazine with the proviso that it would never be made into a film. The manga—titled Nausicaä of the Valley of the Wind—ran from February 1982 to March 1994. The story, as re-printed in the tankōbon volumes, spans seven volumes for a combined total of 1060 pages. Miyazaki drew the episodes primarily in pencil, and it was printed monochrome in sepia-toned ink. Miyazaki resigned from Telecom Animation Film in November 1982.', 'category': 'person'},{'text': \"Following the completion of Nausicaä of the Valley of the Wind's first two volumes, Suzuki and the other editors of Animage encouraged Miyazaki to work on a film adaptation; some documentaries claim he began writing the manga after his film pitch was rejected, but Miyazaki said the manga came first. Miyazaki's imagination was sparked by the mercury poisoning of Minamata Bay and how nature responded and thrived in a poisoned environment, using it to create the film's polluted world. By this time, Miyazaki had moved to the animation studio Topcraft and was finding some of the staff to be unreliable. He eventually decided to bring on several of his previous collaborators for the film's production, including Takahata who would serve as producer. Pre-production began on May 31, 1983; Miyazaki encountered difficulties in creating the screenplay, with only sixteen chapters of the manga to work with. Takahata enlisted experimental and minimalist musician Joe Hisaishi to compose the film's score. Nausicaä of the Valley of the Wind was released on March 11, 1984. It grossed ¥1.48 billion at the box office, and made an additional ¥742 million in distribution income. It is often seen as Miyazaki's pivotal work, cementing his reputation as an animator. It was lauded for its positive portrayal of women, particularly that of main character Nausicaä. Several critics have labeled Nausicaä of the Valley of the Wind as possessing anti-war and feminist themes; Miyazaki argues otherwise, stating that he only wishes to entertain. The successful cooperation on the creation of the manga and the film laid the foundation for other collaborative projects. In April 1984, Miyazaki opened his own office in Suginami Ward, naming it Nibariki.\", 'category': 'person'},{'text': 'Studio Ghibli\\nEarly films (1985–1996)\\nOn June 15, 1985, Miyazaki and Takahata founded the animation production company Studio Ghibli as a subsidiary of Tokuma Shoten. Studio Ghibli\\'s first film was Laputa: Castle in the Sky (1986), directed by Miyazaki. Some of the architecture in the film was also inspired by a Welsh mining town; Miyazaki witnessed the mining strike upon his first visit to Wales in 1984 and admired the miners\\' dedication to their work and community. Laputa was released on August 2, 1986, by the Toei Company. It sold around 775,000 tickets; Miyazaki and Suzuki expressed their disappointment with the film\\'s box office figures. Miyazaki\\'s following film, My Neighbor Totoro, was released alongside Takahata\\'s Grave of the Fireflies in April 1988 to ensure Studio Ghibli\\'s financial status. My Neighbor Totoro features the theme of the relationship between the environment and humanity, showing that harmony is the result of respecting the environment. While the film received critical acclaim, it was commercially unsuccessful at the box office. However, merchandising was successful, and the film was labeled as a cult classic.\\nIn 1987, Studio Ghibli acquired the rights to create a film adaptation of Eiko Kadono\\'s novel Kiki\\'s Delivery Service. Miyazaki\\'s work on My Neighbor Totoro prevented him from directing the adaptation; Sunao Katabuchi was chosen as director, and Nobuyuki Isshiki was hired as script writer. Miyazaki\\'s dissatisfaction of Isshiki\\'s first draft led him to make changes to the project, ultimately taking the role of director. Kadono was unhappy with the differences between the book and the screenplay. Miyazaki and Suzuki visited Kadono and invited her to the studio; she allowed the project to continue. The film was originally intended to be a 60-minute special, but expanded into a feature film after Miyazaki completed the storyboards and screenplay. Kiki\\'s Delivery Service premiered on July 29, 1989. It earned ¥2.15 billion at the box office, and was the highest-grossing film in Japan in 1989.\\nFrom March to May 1989, Miyazaki\\'s manga Hikōtei Jidai was published in the magazine Model Graphix. Miyazaki began production on a 45-minute in-flight film for Japan Airlines based on the manga; Suzuki ultimately extended the film into the feature-length film, titled Porco Rosso, as expectations grew. The outbreak of the Yugoslav Wars in 1991 affected Miyazaki, prompting a more sombre tone for the film; Miyazaki would later refer to the film as \"foolish\", as its mature tones were unsuitable for children. The film featured anti-war themes, which Miyazaki would later revisit. The airline remained a major investor in the film, resulting in its initial premiere as an in-flight film, prior to its theatrical release on July 18, 1992. The film was commercially successful and remained one of the highest-grossing films in Japan for several years.\\nStudio Ghibli set up its headquarters in Koganei, Tokyo in August 1992. In November 1992, two television spots directed by Miyazaki were broadcast by Nippon Television Network (NTV): Sora Iro no Tane, a 90-second spot adapted from the illustrated story Sora Iro no Tane by Rieko Nakagawa and Yuriko Omura; and Nandarou, a series of five advertisements featuring an undefinable creature. Miyazaki designed the storyboards and wrote the screenplay for Whisper of the Heart (1995), directed by Yoshifumi Kondō.', 'category': 'person'},{'text': 'Global emergence (1997–2008)\\nMiyazaki began work on the initial storyboards for Princess Mononoke in August 1994, based on preliminary thoughts and sketches from the late 1970s. While experiencing writer\\'s block during production, Miyazaki accepted a request for the creation of On Your Mark, a music video for the song of the same name by Chage and Aska. In the production of the video, Miyazaki experimented with computer animation to supplement traditional animation. On Your Mark premiered as a short before Whisper of the Heart. Despite the video\\'s popularity, Suzuki said that it was not given \"100 percent\" focus.', 'category': 'person'},{'text': 'In May 1995, Miyazaki took a group of artists and animators to the ancient forests of Yakushima and the mountains of Shirakami-Sanchi, taking photographs and making sketches. The landscapes in the film were inspired by Yakushima. In Princess Mononoke, Miyazaki revisited the ecological and political themes of Nausicaä of the Valley of the Wind. Miyazaki supervised the 144,000 cels in the film, about 80,000 of which were key animation. Princess Mononoke was produced with an estimated budget of ¥2.35 billion (approximately US$23.5 million), making it the most expensive Japanese animated film at the time. Approximately fifteen minutes of the film uses computer animation: about five minutes uses techniques such as 3D rendering, digital composition, and texture mapping; the remaining ten minutes uses digital ink and paint. While the original intention was to digitally paint 5,000 of the film\\'s frames, time constraints doubled this, though it remained below ten percent of the final film.\\nUpon its premiere on July 12, 1997, Princess Mononoke was critically acclaimed, becoming the first animated film to win the Japan Academy Film Prize for Picture of the Year. The film was also commercially successful, becoming the highest-grossing film in Japan for several months. Miramax Films purchased the film\\'s distributions rights for North America; while it was largely unsuccessful at the box office, grossing about US$2.3 million, it was seen as the introduction of Studio Ghibli to global markets. Miyazaki claimed Princess Mononoke would be his final film. Tokuma Shoten merged with Studio Ghibli in June 1997. Miyazaki left Studio Ghibli on January 14, 1998, to create a new studio called Butaya, to be succeeded by Kondō; however, Kondō\\'s death impacted Miyazaki, and he returned to Studio Ghibli on January 16, 1999.\\nMiyazaki\\'s next film was conceived while on vacation at a mountain cabin with his family and five young girls who were family friends. Miyazaki realized that he had not created a film for 10-year-old girls, and set out to do so. He read shōjō manga magazines like Nakayoshi and Ribon for inspiration, but felt they only offered subjects on \"crushes and romance\", which is not what the girls \"held dear in their hearts\". He decided to produce the film about a female heroine whom they could look up to. Production of the film, titled Spirited Away, commenced in 2000 on a budget of ¥1.9 billion (US$15 million). As with Princess Mononoke, the staff experimented with computer animation, but kept the technology at a level to enhance the story, not to \"steal the show\". Spirited Away deals with symbols of human greed, symbolizing the 1980s Japanese asset price bubble, and a liminal journey through the realm of spirits. The film was released on July 20, 2001; it received critical acclaim, and is considered among the greatest films of the 2000s. It won the Japan Academy Film Prize for Picture of the Year, and the Academy Award for Best Animated Feature. The film was also commercially successful, earning ¥30.4 billion (US$289.1 million) at the box office. It became the highest-grossing film in Japan, a record it maintained for almost 20 years. Following the death of Tokuma in September 2000, Miyazaki served as the head of his funeral committee.\\nIn September 2001, Studio Ghibli announced the production of Howl\\'s Moving Castle, based on the novel by Diana Wynne Jones. Mamoru Hosoda of Toei Animation was originally selected to direct the film, but disagreements between Hosoda and Studio Ghibli executives led to the project\\'s abandonment. After six months, Studio Ghibli resurrected the project. Miyazaki was inspired to direct the film upon reading Jones\\' novel, and was struck by the image of a castle moving around the countryside; the novel does not explain how the castle moved, which led to Miyazaki\\'s designs. He traveled to Colmar and Riquewihr in Alsace, France, to study the architecture and the surroundings for the film\\'s setting. Additional inspiration came from the concepts of future technology in Albert Robida\\'s work. It was released on November 20, 2004, and received widespread critical acclaim. The film received the Osella Award for Technical Excellence at the 61st Venice International Film Festival, and was nominated for the Academy Award for Best Animated Feature. In Japan, the film grossed a record $14.5 million in its first week of release. It remains among the highest-grossing films in Japan, with a worldwide gross of over ¥19.3 billion. Miyazaki received the honorary Golden Lion for Lifetime Achievement award at the 62nd Venice International Film Festival in 2005.\\nIn March 2005, Studio Ghibli split from Tokuma Shoten. In the 1980s, Miyazaki had contacted Ursula K. Le Guin expressing interest in producing an adaptation of her Earthsea novels; unaware of Miyazaki\\'s work, Le Guin declined. Upon watching My Neighbor Totoro several years later, Le Guin expressed approval to the concept of the adaptation. She met with Suzuki in August 2005, who wanted Miyazaki\\'s son Goro to direct the film, as Miyazaki had wished to retire. Disappointed that Miyazaki was not directing, but under the impression that he would supervise his son\\'s work, Le Guin approved of the film\\'s production. Miyazaki later publicly opposed and criticized Gorō\\'s appointment as director. Upon Miyazaki\\'s viewing of the film, he wrote a message for his son: \"It was made honestly, so it was good\".\\nMiyazaki designed the covers for several manga novels in 2006, including A Trip to Tynemouth; he also worked as editor, and created a short manga for the book. Miyazaki\\'s next film, Ponyo, began production in May 2006. It was initially inspired by \"The Little Mermaid\" by Hans Christian Andersen, though began to take its own form as production continued. Miyazaki aimed for the film to celebrate the innocence and cheerfulness of a child\\'s universe. He intended for it to only use traditional animation, and was intimately involved with the artwork. He preferred to draw the sea and waves himself, as he enjoyed experimenting. Ponyo features 170,000 frames—a record for Miyazaki. The film\\'s seaside village was inspired by Tomonoura, a town in Setonaikai National Park, where Miyazaki stayed in 2005. The main character, Sōsuke, is based on Gorō. Following its release on July 19, 2008, Ponyo was critically acclaimed, receiving Animation of the Year at the 32nd Japan Academy Film Prize. The film was also a commercial success, earning ¥10 billion (US$93.2 million) in its first month and ¥15.5 billion by the end of 2008, placing it among the highest-grossing films in Japan.', 'category': 'person'},{'text': 'Later films (2009–present)\\nIn early 2009, Miyazaki began writing a manga called Kaze Tachinu (風立ちぬ, The Wind Rises), telling the story of Mitsubishi A6M Zero fighter designer Jiro Horikoshi. The manga was first published in two issues of the Model Graphix magazine, published on February 25 and March 25, 2009. Miyazaki later co-wrote the screenplay for Arrietty (2010) and From Up on Poppy Hill (2011), directed by Hiromasa Yonebayashi and Gorō Miyazaki respectively. Miyazaki wanted his next film to be a sequel to Ponyo, but Suzuki convinced him to instead adapt Kaze Tachinu to film. In November 2012, Studio Ghibli announced the production of The Wind Rises, based on Kaze Tachinu, to be released alongside Takahata\\'s The Tale of the Princess Kaguya.\\nMiyazaki was inspired to create The Wind Rises after reading a quote from Horikoshi: \"All I wanted to do was to make something beautiful\". Several scenes in The Wind Rises were inspired by Tatsuo Hori\\'s novel The Wind Has Risen (風立ちぬ), in which Hori wrote about his life experiences with his fiancée before she died from tuberculosis. The female lead character\\'s name, Naoko Satomi, was borrowed from Hori\\'s novel Naoko (菜穂子). The Wind Rises continues to reflect Miyazaki\\'s pacifist stance, continuing the themes of his earlier works, despite stating that condemning war was not the intention of the film. The film premiered on July 20, 2013, and received critical acclaim; it was named Animation of the Year at the 37th Japan Academy Film Prize, and was nominated for Best Animated Feature at the 86th Academy Awards. It was also commercially successful, grossing ¥11.6 billion (US$110 million) at the Japanese box office, becoming the highest-grossing film in Japan in 2013.\\nIn September 2013, Miyazaki announced that he was retiring from the production of feature films due to his age, but wished to continue working on the displays at the Studio Ghibli Museum. Miyazaki was awarded the Academy Honorary Award at the Governors Awards in November 2014. He developed Boro the Caterpillar, an animated short film which was first discussed during pre-production for Princess Mononoke. It was screened exclusively at the Studio Ghibli Museum in July 2017. Around this time, Miyazaki was working on a manga titled Teppo Samurai. In February 2019, a four-part documentary was broadcast on the NHK network titled 10 Years with Hayao Miyazaki, documenting production of his films in his private studio. In 2019, Miyazaki approved a musical adaptation of Nausicaä of the Valley of the Wind, as it was performed by a kabuki troupe.\\nIn August 2016, Miyazaki proposed a new feature-length film, Kimi-tachi wa Dō Ikiru ka (titled The Boy and the Heron in English), on which he began animation work without receiving official approval. The film opened in Japanese theaters on July 14, 2023. It was preceded by a minimal marketing campaign, forgoing trailers, commercials, and advertisements, a response from Suzuki to his perceived oversaturation of marketing materials in mainstream films. Despite claims that The Boy and the Heron would be Miyazaki\\'s final film, Studio Ghibli vice president Junichi Nishioka said in September 2023 that Miyazaki continued to attend the office daily to plan his next film. Suzuki said he could no longer convince Miyazaki to retire. The Boy and the Heron won Miyazaki his second Academy Award for Best Animated Feature at the 96th Academy Awards, becoming the oldest director to do so; Miyazaki did not attend the show due to his advanced age.', 'category': 'person'}...]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install pyarrow -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/msivanes/miniconda3/envs/rag/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# Enter Lancedb\n",
    "import lancedb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lancedb.pydantic import LanceModel, Vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭───────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.pydantic.LanceModel'</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ─────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">class </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">LanceModel</span><span style=\"font-weight: bold\">()</span> -&gt; <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>:                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">A Pydantic Model base class that can be converted to a LanceDB Table.</span>   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Examples</span>                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">--------</span>                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; import lancedb</span>                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; from lancedb.pydantic import LanceModel, Vector</span>                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt;</span>                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; class </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">LanceModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">:</span>                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     name: str</span>                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     vector: </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Vector</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span>                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; db = </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">lancedb.connect</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"./example\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; table = </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">db.create_table</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"test\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">schema</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.to_arrow_schema</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">())</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">table.add</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">([</span>                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"test\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">vector</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">])</span>                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">])</span>                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">table.search</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">([</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"color: #008080; text-decoration-color: #008080\">., </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"color: #008080; text-decoration-color: #008080\">.</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">])</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.limit</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.to_pydantic</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">'test'</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">vector</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FixedSizeList</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">dim</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">))]</span>                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_computed_fields</span> = <span style=\"font-weight: bold\">{}</span>                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_config</span> = <span style=\"font-weight: bold\">{}</span>                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>           <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_extra</span> = <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">property</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f1169e082c0</span><span style=\"font-weight: bold\">&gt;</span>             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields</span> = <span style=\"font-weight: bold\">{}</span>                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>      <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields_set</span> = <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">property</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f10b812a1b0</span><span style=\"font-weight: bold\">&gt;</span>             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'lancedb.pydantic.LanceModel'\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mclass \u001b[0m\u001b[1;31mLanceModel\u001b[0m\u001b[1m(\u001b[0m\u001b[1m)\u001b[0m -> \u001b[3;35mNone\u001b[0m:                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mA Pydantic Model base class that can be converted to a LanceDB Table.\u001b[0m   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mExamples\u001b[0m                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m--------\u001b[0m                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> import lancedb\u001b[0m                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> from lancedb.pydantic import LanceModel, Vector\u001b[0m                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>>\u001b[0m                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> class \u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mLanceModel\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m:\u001b[0m                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     name: str\u001b[0m                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     vector: \u001b[0m\u001b[1;35mVector\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;36m)\u001b[0m                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> db = \u001b[0m\u001b[1;35mlancedb.connect\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"./example\"\u001b[0m\u001b[1;36m)\u001b[0m                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> table = \u001b[0m\u001b[1;35mdb.create_table\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"test\"\u001b[0m\u001b[36m, \u001b[0m\u001b[33mschema\u001b[0m\u001b[36m=\u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;35m.to_arrow_schema\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m)\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mtable.add\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m[\u001b[0m                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     \u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mname\u001b[0m\u001b[36m=\u001b[0m\u001b[32m\"test\"\u001b[0m\u001b[36m, \u001b[0m\u001b[33mvector\u001b[0m\u001b[36m=\u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m1.0\u001b[0m\u001b[36m, \u001b[0m\u001b[1;36m2.0\u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mtable.search\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[36m., \u001b[0m\u001b[1;36m0\u001b[0m\u001b[36m.\u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;35m.limit\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;35m.to_pydantic\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mTestModel\u001b[0m\u001b[1;36m)\u001b[0m              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m[\u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mname\u001b[0m\u001b[36m=\u001b[0m\u001b[32m'test'\u001b[0m\u001b[36m, \u001b[0m\u001b[33mvector\u001b[0m\u001b[36m=\u001b[0m\u001b[1;35mFixedSizeList\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mdim\u001b[0m\u001b[36m=\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m]\u001b[0m                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;33mmodel_computed_fields\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_config\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m           \u001b[3;33mmodel_extra\u001b[0m = \u001b[1m<\u001b[0m\u001b[1;95mproperty\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f1169e082c0\u001b[0m\u001b[1m>\u001b[0m             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_fields\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m      \u001b[3;33mmodel_fields_set\u001b[0m = \u001b[1m<\u001b[0m\u001b[1;95mproperty\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f10b812a1b0\u001b[0m\u001b[1m>\u001b[0m             \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(LanceModel, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">function</span><span style=\"color: #000000; text-decoration-color: #000000\"> Vector at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f10b8db2160</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">def </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Vector</span><span style=\"font-weight: bold\">(</span>dim: <span style=\"color: #008000; text-decoration-color: #008000\">'int'</span>, value_type: <span style=\"color: #008000; text-decoration-color: #008000\">'pa.DataType'</span> = <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">DataType</span><span style=\"font-weight: bold\">(</span>float<span style=\"font-weight: bold\">))</span> -&gt; <span style=\"color: #008000; text-decoration-color: #008000\">'Type[FixedSizeListMixin]'</span>: <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Pydantic Vector Type.</span>                                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">!!! warning</span>                                                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    Experimental feature.</span>                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Parameters</span>                                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">----------</span>                                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">dim : int</span>                                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    The dimension of the vector.</span>                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">value_type : pyarrow.DataType, optional</span>                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    The value type of the vector, by default </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.float32</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">()</span>                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Examples</span>                                                                                           <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">--------</span>                                                                                           <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; import pydantic</span>                                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; from lancedb.pydantic import Vector</span>                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span>                                                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; class </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">MyModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">pydantic.BaseModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">:</span>                                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     id: int</span>                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     url: str</span>                                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     embeddings: </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Vector</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">768</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; schema = </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pydantic_to_schema</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">MyModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                           <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; assert schema == </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.schema</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">([</span>                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.field</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"id\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.int64</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">()</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">,</span>                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.field</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"url\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.utf8</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">()</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">,</span>                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.field</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"embeddings\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.list_</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">pa.float32</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">()</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">768</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">])</span>                                                                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span><span style=\"font-style: italic\"> attribute(s) not shown.</span> Run <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">inspect</span><span style=\"font-weight: bold\">(</span>inspect<span style=\"font-weight: bold\">)</span> for options.                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mfunction\u001b[0m\u001b[39m Vector at \u001b[0m\u001b[1;36m0x7f10b8db2160\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m───────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mdef \u001b[0m\u001b[1;31mVector\u001b[0m\u001b[1m(\u001b[0mdim: \u001b[32m'int'\u001b[0m, value_type: \u001b[32m'pa.DataType'\u001b[0m = \u001b[1;35mDataType\u001b[0m\u001b[1m(\u001b[0mfloat\u001b[1m)\u001b[0m\u001b[1m)\u001b[0m -> \u001b[32m'Type\u001b[0m\u001b[32m[\u001b[0m\u001b[32mFixedSizeListMixin\u001b[0m\u001b[32m]\u001b[0m\u001b[32m'\u001b[0m: \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mPydantic Vector Type.\u001b[0m                                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m!!! warning\u001b[0m                                                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    Experimental feature.\u001b[0m                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mParameters\u001b[0m                                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m----------\u001b[0m                                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mdim : int\u001b[0m                                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    The dimension of the vector.\u001b[0m                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mvalue_type : pyarrow.DataType, optional\u001b[0m                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    The value type of the vector, by default \u001b[0m\u001b[1;35mpa.float32\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mExamples\u001b[0m                                                                                           \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m--------\u001b[0m                                                                                           \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> import pydantic\u001b[0m                                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> from lancedb.pydantic import Vector\u001b[0m                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m                                                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> class \u001b[0m\u001b[1;35mMyModel\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mpydantic.BaseModel\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m:\u001b[0m                                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     id: int\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     url: str\u001b[0m                                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     embeddings: \u001b[0m\u001b[1;35mVector\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m768\u001b[0m\u001b[1;36m)\u001b[0m                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> schema = \u001b[0m\u001b[1;35mpydantic_to_schema\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mMyModel\u001b[0m\u001b[1;36m)\u001b[0m                                                           \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> assert schema == \u001b[0m\u001b[1;35mpa.schema\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m[\u001b[0m                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     \u001b[0m\u001b[1;35mpa.field\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"id\"\u001b[0m\u001b[36m, \u001b[0m\u001b[1;35mpa.int64\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m, \u001b[0m\u001b[3;91mFalse\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m,\u001b[0m                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     \u001b[0m\u001b[1;35mpa.field\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"url\"\u001b[0m\u001b[36m, \u001b[0m\u001b[1;35mpa.utf8\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m, \u001b[0m\u001b[3;91mFalse\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m,\u001b[0m                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     \u001b[0m\u001b[1;35mpa.field\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"embeddings\"\u001b[0m\u001b[36m, \u001b[0m\u001b[1;35mpa.list_\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;35mpa.float32\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m, \u001b[0m\u001b[1;36m768\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m, \u001b[0m\u001b[3;91mFalse\u001b[0m\u001b[1;36m)\u001b[0m                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m                                                                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m37\u001b[0m\u001b[3m attribute(s) not shown.\u001b[0m Run \u001b[1;35minspect\u001b[0m\u001b[1m(\u001b[0minspect\u001b[1m)\u001b[0m for options.                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(Vector, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lancedb.embeddings import get_registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.embeddings.registry.EmbeddingFunctionRegistry'</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">This is a singleton class used to register embedding functions</span>                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">and fetch them by name. It also handles serializing and deserializing.</span>               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">You can implement your own embedding function by subclassing EmbeddingFunction</span>       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">or TextEmbeddingFunction and registering it with the registry.</span>                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">NOTE: Here TEXT is a type alias for Union</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080\">str, List</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080\">str</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">]</span><span style=\"color: #008080; text-decoration-color: #008080\">, pa.Array,</span>                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">      pa.ChunkedArray, np.ndarray</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">]</span>                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Examples</span>                                                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">--------</span>                                                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; registry = </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">EmbeddingFunctionRegistry.get_instance</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">()</span>                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; @</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">registry.register</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"my-embedding-function\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\"> class </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">MyEmbeddingFunction</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">EmbeddingFunction</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">:</span>                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     def </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ndims</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">self</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\"> -&gt; int:</span>                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">         return </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">128</span>                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span>                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     def </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">compute_query_embeddings</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">self, query: str, *args, **kwargs</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">:</span>             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">         return </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">self.compute_source_embeddings</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">query, *args, **kwargs</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span>                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     def </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">compute_source_embeddings</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">self, texts, *args, **kwargs</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">:</span>                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">         return </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">np.random.rand</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">self.ndims</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">())</span><span style=\"color: #008080; text-decoration-color: #008080\"> for _ in </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">range</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">len</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">texts</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">))]</span>         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span>                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">registry.get</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"my-embedding-function\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.embeddings.registry.MyEmbeddingFunction'</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">&gt;</span>                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">╭──────────────────────────────────────────────────────────────────────────────────╮</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">lancedb.embeddings.registry.EmbeddingFunctionRegistry</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f10b7761a10</span><span style=\"font-weight: bold\">&gt;</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">╰──────────────────────────────────────────────────────────────────────────────────╯</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span><span style=\"font-style: italic\"> attribute(s) not shown.</span> Run <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">inspect</span><span style=\"font-weight: bold\">(</span>inspect<span style=\"font-weight: bold\">)</span> for options.                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰──────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m─────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'lancedb.embeddings.registry.EmbeddingFunctionRegistry'\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mThis is a singleton class used to register embedding functions\u001b[0m                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mand fetch them by name. It also handles serializing and deserializing.\u001b[0m               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mYou can implement your own embedding function by subclassing EmbeddingFunction\u001b[0m       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mor TextEmbeddingFunction and registering it with the registry.\u001b[0m                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mNOTE: Here TEXT is a type alias for Union\u001b[0m\u001b[1;36m[\u001b[0m\u001b[36mstr, List\u001b[0m\u001b[1;36m[\u001b[0m\u001b[36mstr\u001b[0m\u001b[1;36m]\u001b[0m\u001b[36m, pa.Array,\u001b[0m                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m      pa.ChunkedArray, np.ndarray\u001b[0m\u001b[1;36m]\u001b[0m                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mExamples\u001b[0m                                                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m--------\u001b[0m                                                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> registry = \u001b[0m\u001b[1;35mEmbeddingFunctionRegistry.get_instance\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> @\u001b[0m\u001b[1;35mregistry.register\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"my-embedding-function\"\u001b[0m\u001b[1;36m)\u001b[0m                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m class \u001b[0m\u001b[1;35mMyEmbeddingFunction\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mEmbeddingFunction\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m:\u001b[0m                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     def \u001b[0m\u001b[1;35mndims\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mself\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m -> int:\u001b[0m                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m         return \u001b[0m\u001b[1;36m128\u001b[0m                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     def \u001b[0m\u001b[1;35mcompute_query_embeddings\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mself, query: str, *args, **kwargs\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m:\u001b[0m             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m         return \u001b[0m\u001b[1;35mself.compute_source_embeddings\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mquery, *args, **kwargs\u001b[0m\u001b[1;36m)\u001b[0m            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     def \u001b[0m\u001b[1;35mcompute_source_embeddings\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mself, texts, *args, **kwargs\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m:\u001b[0m                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m         return \u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;35mnp.random.rand\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;35mself.ndims\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m for _ in \u001b[0m\u001b[1;35mrange\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;35mlen\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mtexts\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m]\u001b[0m         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mregistry.get\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"my-embedding-function\"\u001b[0m\u001b[1;36m)\u001b[0m                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'lancedb.embeddings.registry.MyEmbeddingFunction'\u001b[0m\u001b[1;36m>\u001b[0m                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m╭──────────────────────────────────────────────────────────────────────────────────╮\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m│\u001b[0m \u001b[1m<\u001b[0m\u001b[1;95mlancedb.embeddings.registry.EmbeddingFunctionRegistry\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f10b7761a10\u001b[0m\u001b[1m>\u001b[0m \u001b[32m│\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m╰──────────────────────────────────────────────────────────────────────────────────╯\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m28\u001b[0m\u001b[3m attribute(s) not shown.\u001b[0m Run \u001b[1;35minspect\u001b[0m\u001b[1m(\u001b[0minspect\u001b[1m)\u001b[0m for options.                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰──────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(get_registry(), help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_registry = get_registry().get('sentence-transformers')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings'</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">class </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">SentenceTransformerEmbeddings</span><span style=\"font-weight: bold\">(</span>*, max_retries: int = <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span>, name: str = <span style=\"color: #008000; text-decoration-color: #008000\">'all-MiniLM-L6-v2'</span>, device: str =      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">'cpu'</span>, normalize: bool = <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>:                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">An embedding function that uses the sentence-transformers library</span>                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff; text-decoration: underline\">https://huggingface.co/sentence-transformers</span>                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>       <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">embedding_model</span> = <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">property</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f10b7777b50</span><span style=\"font-weight: bold\">&gt;</span>                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_computed_fields</span> = <span style=\"font-weight: bold\">{}</span>                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_config</span> = <span style=\"font-weight: bold\">{}</span>                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>           <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_extra</span> = <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">property</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f1169e082c0</span><span style=\"font-weight: bold\">&gt;</span>                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields</span> = <span style=\"font-weight: bold\">{</span>                                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'max_retries'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">int</span>, <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"font-weight: bold\">)</span>,                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'name'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span>                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">str</span>,                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>,                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'all-MiniLM-L6-v2'</span>                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"font-weight: bold\">)</span>,                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">str</span>, <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cpu'</span><span style=\"font-weight: bold\">)</span>,                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'normalize'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">bool</span>, <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">)</span>               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                         <span style=\"font-weight: bold\">}</span>                                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>      <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields_set</span> = <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">property</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f10b812a1b0</span><span style=\"font-weight: bold\">&gt;</span>                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings'\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m───────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mclass \u001b[0m\u001b[1;31mSentenceTransformerEmbeddings\u001b[0m\u001b[1m(\u001b[0m*, max_retries: int = \u001b[1;36m7\u001b[0m, name: str = \u001b[32m'all-MiniLM-L6-v2'\u001b[0m, device: str =      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m'cpu'\u001b[0m, normalize: bool = \u001b[3;92mTrue\u001b[0m\u001b[1m)\u001b[0m -> \u001b[3;35mNone\u001b[0m:                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mAn embedding function that uses the sentence-transformers library\u001b[0m                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[4;94mhttps://huggingface.co/sentence-transformers\u001b[0m                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m       \u001b[3;33membedding_model\u001b[0m = \u001b[1m<\u001b[0m\u001b[1;95mproperty\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f10b7777b50\u001b[0m\u001b[1m>\u001b[0m                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;33mmodel_computed_fields\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_config\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m           \u001b[3;33mmodel_extra\u001b[0m = \u001b[1m<\u001b[0m\u001b[1;95mproperty\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f1169e082c0\u001b[0m\u001b[1m>\u001b[0m                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_fields\u001b[0m = \u001b[1m{\u001b[0m                                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'max_retries'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m\u001b[33mannotation\u001b[0m=\u001b[35mint\u001b[0m, \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m, \u001b[33mdefault\u001b[0m=\u001b[1;36m7\u001b[0m\u001b[1m)\u001b[0m,                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'name'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mannotation\u001b[0m=\u001b[35mstr\u001b[0m,                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m,                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mdefault\u001b[0m=\u001b[32m'all-MiniLM-L6-v2'\u001b[0m                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[1m)\u001b[0m,                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'device'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m\u001b[33mannotation\u001b[0m=\u001b[35mstr\u001b[0m, \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m, \u001b[33mdefault\u001b[0m=\u001b[32m'cpu'\u001b[0m\u001b[1m)\u001b[0m,                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'normalize'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m\u001b[33mannotation\u001b[0m=\u001b[35mbool\u001b[0m, \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m, \u001b[33mdefault\u001b[0m=\u001b[3;92mTrue\u001b[0m\u001b[1m)\u001b[0m               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                         \u001b[1m}\u001b[0m                                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m      \u001b[3;33mmodel_fields_set\u001b[0m = \u001b[1m<\u001b[0m\u001b[1;95mproperty\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f10b812a1b0\u001b[0m\u001b[1m>\u001b[0m                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(model_registry, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─ </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">bound</span><span style=\"color: #000000; text-decoration-color: #000000\"> method EmbeddingFunction.create of &lt;class </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.embeddings.sentence_transformers.SentenceTransformer</span><span style=\"color: #000080; text-decoration-color: #000080\">─╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">def </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">EmbeddingFunction.create</span><span style=\"font-weight: bold\">(</span>**kwargs<span style=\"font-weight: bold\">)</span>:                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Create an instance of the embedding function</span>                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span><span style=\"font-style: italic\"> attribute(s) not shown.</span> Run <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">inspect</span><span style=\"font-weight: bold\">(</span>inspect<span style=\"font-weight: bold\">)</span> for options.                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mbound\u001b[0m\u001b[39m method EmbeddingFunction.create of <class \u001b[0m\u001b[32m'lancedb.embeddings.sentence_transformers.SentenceTransformer\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mdef \u001b[0m\u001b[1;31mEmbeddingFunction.create\u001b[0m\u001b[1m(\u001b[0m**kwargs\u001b[1m)\u001b[0m:                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mCreate an instance of the embedding function\u001b[0m                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m27\u001b[0m\u001b[3m attribute(s) not shown.\u001b[0m Run \u001b[1;35minspect\u001b[0m\u001b[1m(\u001b[0minspect\u001b[1m)\u001b[0m for options.                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(model_registry.create, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model_registry.create(name='BAAI/bge-small-en-v1.5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SentenceTransformerEmbeddings(max_retries=7, name='BAAI/bge-small-en-v1.5', device='cpu', normalize=True)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/msivanes/miniconda3/envs/rag/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings'</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">An embedding function that uses the sentence-transformers library</span>                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff; text-decoration: underline\">https://huggingface.co/sentence-transformers</span>                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SentenceTransformerEmbeddings</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">max_retries</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span>, <span style=\"color: #808000; text-decoration-color: #808000\">name</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'BAAI/bge-small-en-v1.5'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">device</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cpu'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">normalize</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">)</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">device</span> = <span style=\"color: #008000; text-decoration-color: #008000\">'cpu'</span>                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>           <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">max_retries</span> = <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span>                                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_computed_fields</span> = <span style=\"font-weight: bold\">{}</span>                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_config</span> = <span style=\"font-weight: bold\">{}</span>                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>           <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_extra</span> = <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields</span> = <span style=\"font-weight: bold\">{</span>                                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'max_retries'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">int</span>, <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"font-weight: bold\">)</span>,              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'name'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span>                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">str</span>,                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>,                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'all-MiniLM-L6-v2'</span>                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"font-weight: bold\">)</span>,                                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">str</span>, <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cpu'</span><span style=\"font-weight: bold\">)</span>,               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'normalize'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">bool</span>, <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #808000; text-decoration-color: #808000\">default</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">)</span>             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                         <span style=\"font-weight: bold\">}</span>                                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>      <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields_set</span> = <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'name'</span><span style=\"font-weight: bold\">}</span>                                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                  <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">name</span> = <span style=\"color: #008000; text-decoration-color: #008000\">'BAAI/bge-small-en-v1.5'</span>                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>             <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">normalize</span> = <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m─────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings'\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mAn embedding function that uses the sentence-transformers library\u001b[0m                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[4;94mhttps://huggingface.co/sentence-transformers\u001b[0m                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m│\u001b[0m \u001b[1;35mSentenceTransformerEmbeddings\u001b[0m\u001b[1m(\u001b[0m\u001b[33mmax_retries\u001b[0m=\u001b[1;36m7\u001b[0m, \u001b[33mname\u001b[0m=\u001b[32m'BAAI/bge-small-en-v1.5'\u001b[0m, \u001b[33mdevice\u001b[0m=\u001b[32m'cpu'\u001b[0m, \u001b[33mnormalize\u001b[0m=\u001b[3;92mTrue\u001b[0m\u001b[1m)\u001b[0m \u001b[32m│\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                \u001b[3;33mdevice\u001b[0m = \u001b[32m'cpu'\u001b[0m                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m           \u001b[3;33mmax_retries\u001b[0m = \u001b[1;36m7\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;33mmodel_computed_fields\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_config\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m           \u001b[3;33mmodel_extra\u001b[0m = \u001b[3;35mNone\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_fields\u001b[0m = \u001b[1m{\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'max_retries'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m\u001b[33mannotation\u001b[0m=\u001b[35mint\u001b[0m, \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m, \u001b[33mdefault\u001b[0m=\u001b[1;36m7\u001b[0m\u001b[1m)\u001b[0m,              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'name'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mannotation\u001b[0m=\u001b[35mstr\u001b[0m,                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m,                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mdefault\u001b[0m=\u001b[32m'all-MiniLM-L6-v2'\u001b[0m                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[1m)\u001b[0m,                                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'device'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m\u001b[33mannotation\u001b[0m=\u001b[35mstr\u001b[0m, \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m, \u001b[33mdefault\u001b[0m=\u001b[32m'cpu'\u001b[0m\u001b[1m)\u001b[0m,               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'normalize'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m\u001b[33mannotation\u001b[0m=\u001b[35mbool\u001b[0m, \u001b[33mrequired\u001b[0m=\u001b[3;91mFalse\u001b[0m, \u001b[33mdefault\u001b[0m=\u001b[3;92mTrue\u001b[0m\u001b[1m)\u001b[0m             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                         \u001b[1m}\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m      \u001b[3;33mmodel_fields_set\u001b[0m = \u001b[1m{\u001b[0m\u001b[32m'name'\u001b[0m\u001b[1m}\u001b[0m                                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                  \u001b[3;33mname\u001b[0m = \u001b[32m'BAAI/bge-small-en-v1.5'\u001b[0m                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m             \u001b[3;33mnormalize\u001b[0m = \u001b[3;92mTrue\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(model, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─ </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">bound</span><span style=\"color: #000000; text-decoration-color: #000000\"> method EmbeddingFunction.SourceField of </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SentenceTransformerEmbeddings</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">max_retries</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">'BAAI/bge-sma</span><span style=\"color: #000080; text-decoration-color: #000080\">─╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">def </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">EmbeddingFunction.SourceField</span><span style=\"font-weight: bold\">(</span>**kwargs<span style=\"font-weight: bold\">)</span>:                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Creates a pydantic Field that can automatically annotate</span>                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">the source column for this embedding function</span>                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span><span style=\"font-style: italic\"> attribute(s) not shown.</span> Run <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">inspect</span><span style=\"font-weight: bold\">(</span>inspect<span style=\"font-weight: bold\">)</span> for options.                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mbound\u001b[0m\u001b[39m method EmbeddingFunction.SourceField of \u001b[0m\u001b[1;35mSentenceTransformerEmbeddings\u001b[0m\u001b[1;39m(\u001b[0m\u001b[33mmax_retries\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m7\u001b[0m\u001b[39m, \u001b[0m\u001b[33mname\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'BAAI/bge-sma\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mdef \u001b[0m\u001b[1;31mEmbeddingFunction.SourceField\u001b[0m\u001b[1m(\u001b[0m**kwargs\u001b[1m)\u001b[0m:                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mCreates a pydantic Field that can automatically annotate\u001b[0m                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mthe source column for this embedding function\u001b[0m                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m27\u001b[0m\u001b[3m attribute(s) not shown.\u001b[0m Run \u001b[1;35minspect\u001b[0m\u001b[1m(\u001b[0minspect\u001b[1m)\u001b[0m for options.                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(model.SourceField, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─ </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">bound</span><span style=\"color: #000000; text-decoration-color: #000000\"> method EmbeddingFunction.VectorField of </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SentenceTransformerEmbeddings</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">max_retries</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">'BAAI/bge-sma</span><span style=\"color: #000080; text-decoration-color: #000080\">─╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">def </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">EmbeddingFunction.VectorField</span><span style=\"font-weight: bold\">(</span>**kwargs<span style=\"font-weight: bold\">)</span>:                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Creates a pydantic Field that can automatically annotate</span>                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">the target vector column for this embedding function</span>                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span><span style=\"font-style: italic\"> attribute(s) not shown.</span> Run <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">inspect</span><span style=\"font-weight: bold\">(</span>inspect<span style=\"font-weight: bold\">)</span> for options.                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mbound\u001b[0m\u001b[39m method EmbeddingFunction.VectorField of \u001b[0m\u001b[1;35mSentenceTransformerEmbeddings\u001b[0m\u001b[1;39m(\u001b[0m\u001b[33mmax_retries\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m7\u001b[0m\u001b[39m, \u001b[0m\u001b[33mname\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'BAAI/bge-sma\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mdef \u001b[0m\u001b[1;31mEmbeddingFunction.VectorField\u001b[0m\u001b[1m(\u001b[0m**kwargs\u001b[1m)\u001b[0m:                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mCreates a pydantic Field that can automatically annotate\u001b[0m                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mthe target vector column for this embedding function\u001b[0m                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m27\u001b[0m\u001b[3m attribute(s) not shown.\u001b[0m Run \u001b[1;35minspect\u001b[0m\u001b[1m(\u001b[0minspect\u001b[1m)\u001b[0m for options.                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(model.VectorField, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Document(LanceModel):\n",
    "    text: str = model.SourceField()\n",
    "    category: str\n",
    "    vector: Vector(384) = model.VectorField()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭──────────────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'__main__.Document'</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ─────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">class </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Document</span><span style=\"font-weight: bold\">(</span>*, text: str, category: str, vector: lancedb.pydantic.Vector.<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">locals</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;.FixedSizeList</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">)</span><span style=\"color: #000000; text-decoration-color: #000000\"> -</span><span style=\"font-weight: bold\">&gt;</span> <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>: <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">A Pydantic Model base class that can be converted to a LanceDB Table.</span>                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Examples</span>                                                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">--------</span>                                                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; import lancedb</span>                                                                                           <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; from lancedb.pydantic import LanceModel, Vector</span>                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt;</span>                                                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; class </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">LanceModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">:</span>                                                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     name: str</span>                                                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     vector: </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Vector</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span>                                                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; db = </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">lancedb.connect</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"./example\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; table = </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">db.create_table</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"test\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">schema</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.to_arrow_schema</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">())</span>                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">table.add</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">([</span>                                                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">     </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"test\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">vector</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">])</span>                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">])</span>                                                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">table.search</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">([</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"color: #008080; text-decoration-color: #008080\">., </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"color: #008080; text-decoration-color: #008080\">.</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">])</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.limit</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.to_pydantic</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TestModel</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">'test'</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">vector</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FixedSizeList</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">dim</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">))]</span>                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_computed_fields</span> = <span style=\"font-weight: bold\">{}</span>                                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_config</span> = <span style=\"font-weight: bold\">{}</span>                                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>           <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_extra</span> = <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">property</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f1169e082c0</span><span style=\"font-weight: bold\">&gt;</span>                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>          <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields</span> = <span style=\"font-weight: bold\">{</span>                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span>                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">str</span>,                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>,                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">json_schema_extra</span>=<span style=\"font-weight: bold\">{</span>                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                     <span style=\"color: #008000; text-decoration-color: #008000\">'source_column_for'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SentenceTransformerEmbeddings</span><span style=\"font-weight: bold\">(</span>                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">max_retries</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span>,                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">name</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'BAAI/bge-small-en-v1.5'</span>,                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">device</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cpu'</span>,                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">normalize</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                     <span style=\"font-weight: bold\">)</span>                                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"font-weight: bold\">}</span>                                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"font-weight: bold\">)</span>,                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'category'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">str</span>, <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">)</span>,                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"color: #008000; text-decoration-color: #008000\">'vector'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FieldInfo</span><span style=\"font-weight: bold\">(</span>                                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">annotation</span>=<span style=\"color: #800080; text-decoration-color: #800080\">Vector</span>.<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">locals</span><span style=\"font-weight: bold\">&gt;</span>.FixedSizeList,                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">required</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>,                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"color: #808000; text-decoration-color: #808000\">json_schema_extra</span>=<span style=\"font-weight: bold\">{</span>                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                     <span style=\"color: #008000; text-decoration-color: #008000\">'vector_column_for'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SentenceTransformerEmbeddings</span><span style=\"font-weight: bold\">(</span>                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">max_retries</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span>,                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">name</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'BAAI/bge-small-en-v1.5'</span>,                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">device</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cpu'</span>,                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                         <span style=\"color: #808000; text-decoration-color: #808000\">normalize</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                     <span style=\"font-weight: bold\">)</span>                                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                 <span style=\"font-weight: bold\">}</span>                                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                             <span style=\"font-weight: bold\">)</span>                                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                         <span style=\"font-weight: bold\">}</span>                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>      <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">model_fields_set</span> = <span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">property</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x7f10b812a1b0</span><span style=\"font-weight: bold\">&gt;</span>                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m───────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'__main__.Document'\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mclass \u001b[0m\u001b[1;31mDocument\u001b[0m\u001b[1m(\u001b[0m*, text: str, category: str, vector: lancedb.pydantic.Vector.\u001b[1m<\u001b[0m\u001b[1;95mlocals\u001b[0m\u001b[39m>.FixedSizeList\u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m -\u001b[0m\u001b[1m>\u001b[0m \u001b[3;35mNone\u001b[0m: \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mA Pydantic Model base class that can be converted to a LanceDB Table.\u001b[0m                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mExamples\u001b[0m                                                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m--------\u001b[0m                                                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> import lancedb\u001b[0m                                                                                           \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> from lancedb.pydantic import LanceModel, Vector\u001b[0m                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>>\u001b[0m                                                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> class \u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mLanceModel\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m:\u001b[0m                                                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     name: str\u001b[0m                                                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     vector: \u001b[0m\u001b[1;35mVector\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;36m)\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m                                                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> db = \u001b[0m\u001b[1;35mlancedb.connect\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"./example\"\u001b[0m\u001b[1;36m)\u001b[0m                                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> table = \u001b[0m\u001b[1;35mdb.create_table\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"test\"\u001b[0m\u001b[36m, \u001b[0m\u001b[33mschema\u001b[0m\u001b[36m=\u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;35m.to_arrow_schema\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m)\u001b[0m                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mtable.add\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m[\u001b[0m                                                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m     \u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mname\u001b[0m\u001b[36m=\u001b[0m\u001b[32m\"test\"\u001b[0m\u001b[36m, \u001b[0m\u001b[33mvector\u001b[0m\u001b[36m=\u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m1.0\u001b[0m\u001b[36m, \u001b[0m\u001b[1;36m2.0\u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m                                                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mtable.search\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[36m., \u001b[0m\u001b[1;36m0\u001b[0m\u001b[36m.\u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;35m.limit\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;35m.to_pydantic\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mTestModel\u001b[0m\u001b[1;36m)\u001b[0m                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m[\u001b[0m\u001b[1;35mTestModel\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mname\u001b[0m\u001b[36m=\u001b[0m\u001b[32m'test'\u001b[0m\u001b[36m, \u001b[0m\u001b[33mvector\u001b[0m\u001b[36m=\u001b[0m\u001b[1;35mFixedSizeList\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mdim\u001b[0m\u001b[36m=\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m]\u001b[0m                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;33mmodel_computed_fields\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_config\u001b[0m = \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m                                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m           \u001b[3;33mmodel_extra\u001b[0m = \u001b[1m<\u001b[0m\u001b[1;95mproperty\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f1169e082c0\u001b[0m\u001b[1m>\u001b[0m                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m          \u001b[3;33mmodel_fields\u001b[0m = \u001b[1m{\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'text'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mannotation\u001b[0m=\u001b[35mstr\u001b[0m,                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mrequired\u001b[0m=\u001b[3;92mTrue\u001b[0m,                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mjson_schema_extra\u001b[0m=\u001b[1m{\u001b[0m                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                     \u001b[32m'source_column_for'\u001b[0m: \u001b[1;35mSentenceTransformerEmbeddings\u001b[0m\u001b[1m(\u001b[0m                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mmax_retries\u001b[0m=\u001b[1;36m7\u001b[0m,                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mname\u001b[0m=\u001b[32m'BAAI/bge-small-en-v1.5'\u001b[0m,                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mdevice\u001b[0m=\u001b[32m'cpu'\u001b[0m,                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mnormalize\u001b[0m=\u001b[3;92mTrue\u001b[0m                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                     \u001b[1m)\u001b[0m                                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[1m}\u001b[0m                                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[1m)\u001b[0m,                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'category'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m\u001b[33mannotation\u001b[0m=\u001b[35mstr\u001b[0m, \u001b[33mrequired\u001b[0m=\u001b[3;92mTrue\u001b[0m\u001b[1m)\u001b[0m,                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[32m'vector'\u001b[0m: \u001b[1;35mFieldInfo\u001b[0m\u001b[1m(\u001b[0m                                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mannotation\u001b[0m=\u001b[35mVector\u001b[0m.\u001b[1m<\u001b[0m\u001b[1;95mlocals\u001b[0m\u001b[1m>\u001b[0m.FixedSizeList,                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mrequired\u001b[0m=\u001b[3;92mTrue\u001b[0m,                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[33mjson_schema_extra\u001b[0m=\u001b[1m{\u001b[0m                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                     \u001b[32m'vector_column_for'\u001b[0m: \u001b[1;35mSentenceTransformerEmbeddings\u001b[0m\u001b[1m(\u001b[0m                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mmax_retries\u001b[0m=\u001b[1;36m7\u001b[0m,                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mname\u001b[0m=\u001b[32m'BAAI/bge-small-en-v1.5'\u001b[0m,                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mdevice\u001b[0m=\u001b[32m'cpu'\u001b[0m,                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                         \u001b[33mnormalize\u001b[0m=\u001b[3;92mTrue\u001b[0m                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                     \u001b[1m)\u001b[0m                                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                 \u001b[1m}\u001b[0m                                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                             \u001b[1m)\u001b[0m                                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                         \u001b[1m}\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m      \u001b[3;33mmodel_fields_set\u001b[0m = \u001b[1m<\u001b[0m\u001b[1;95mproperty\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x7f10b812a1b0\u001b[0m\u001b[1m>\u001b[0m                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(Document, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = lancedb.connect('.my_db')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanceDBConnection(/home/msivanes/Documents/0Inbox/bclavie-beyond-basics-for-retrieval/.my_db)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭───────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.db.LanceDBConnection'</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ─────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">A connection to a LanceDB database.</span>                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Parameters</span>                                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">----------</span>                                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">uri: str or Path</span>                                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    The root uri of the database.</span>                                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">read_consistency_interval: timedelta, default </span><span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    The interval at which to check for updates to the table from other</span>                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    processes. If </span><span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span><span style=\"color: #008080; text-decoration-color: #008080\">, then consistency is not checked. For performance</span>                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    reasons, this is the default. For strong consistency, set this to</span>                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    zero seconds. Then every read will check for updates from other</span>                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    processes. As a compromise, you can set this to a non-zero timedelta</span>                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    for eventual consistency. If more than that interval has passed since</span>                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    the last check, then the table will be checked for updates. Note: this</span>                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    consistency only applies to read operations. Write operations are</span>                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    always consistent.</span>                                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Examples</span>                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">--------</span>                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; import lancedb</span>                                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; db = </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">lancedb.connect</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"./.lancedb\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">db.create_table</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"my_table\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">data</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[{</span><span style=\"color: #008000; text-decoration-color: #008000\">\"vector\"</span><span style=\"color: #008080; text-decoration-color: #008080\">: </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">]</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008000; text-decoration-color: #008000\">\"b\"</span><span style=\"color: #008080; text-decoration-color: #008080\">: </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">}</span><span style=\"color: #008080; text-decoration-color: #008080\">,</span>                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">                                   </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">\"vector\"</span><span style=\"color: #008080; text-decoration-color: #008080\">: </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.5</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">]</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008000; text-decoration-color: #008000\">\"b\"</span><span style=\"color: #008080; text-decoration-color: #008080\">: </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">}])</span>                                   <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">LanceTable</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">connection</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"my_table\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">db.create_table</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"another_table\"</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">data</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[{</span><span style=\"color: #008000; text-decoration-color: #008000\">\"vector\"</span><span style=\"color: #008080; text-decoration-color: #008080\">: </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.4</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.4</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">]</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008000; text-decoration-color: #008000\">\"b\"</span><span style=\"color: #008080; text-decoration-color: #008080\">: </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">}])</span>                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">LanceTable</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">connection</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"another_table\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">sorted</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">db.table_names</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">())</span>                                                                             <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'another_table'</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #008000; text-decoration-color: #008000\">'my_table'</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">]</span>                                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">len</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">db</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>                                                                                                        <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; db</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">\"my_table\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">]</span>                                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">LanceTable</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">connection</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span><span style=\"color: #008080; text-decoration-color: #008080\">, </span><span style=\"color: #808000; text-decoration-color: #808000\">name</span><span style=\"color: #008080; text-decoration-color: #008080\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"my_table\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                              <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #008000; text-decoration-color: #008000\">\"my_table\"</span><span style=\"color: #008080; text-decoration-color: #008080\"> in db</span>                                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>                                                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">db.drop_table</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"my_table\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                                            <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">&gt;&gt;&gt; </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">db.drop_table</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">\"another_table\"</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">LanceDBConnection</span><span style=\"font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">/home/msivanes/Documents/0Inbox/bclavie-beyond-basics-for-retrieval/</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">.my_db</span><span style=\"font-weight: bold\">)</span>        <span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008000; text-decoration-color: #008000\">╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">read_consistency_interval</span> = <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>                                                                         <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000; font-style: italic\">uri</span> = <span style=\"color: #008000; text-decoration-color: #008000\">'/home/msivanes/Documents/0Inbox/bclavie-beyond-basics-for-retrieval/.my_db'</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'lancedb.db.LanceDBConnection'\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mA connection to a LanceDB database.\u001b[0m                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mParameters\u001b[0m                                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m----------\u001b[0m                                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36muri: str or Path\u001b[0m                                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    The root uri of the database.\u001b[0m                                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mread_consistency_interval: timedelta, default \u001b[0m\u001b[3;35mNone\u001b[0m                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    The interval at which to check for updates to the table from other\u001b[0m                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    processes. If \u001b[0m\u001b[3;35mNone\u001b[0m\u001b[36m, then consistency is not checked. For performance\u001b[0m                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    reasons, this is the default. For strong consistency, set this to\u001b[0m                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    zero seconds. Then every read will check for updates from other\u001b[0m                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    processes. As a compromise, you can set this to a non-zero timedelta\u001b[0m                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    for eventual consistency. If more than that interval has passed since\u001b[0m                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    the last check, then the table will be checked for updates. Note: this\u001b[0m                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    consistency only applies to read operations. Write operations are\u001b[0m                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    always consistent.\u001b[0m                                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mExamples\u001b[0m                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m--------\u001b[0m                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> import lancedb\u001b[0m                                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> db = \u001b[0m\u001b[1;35mlancedb.connect\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"./.lancedb\"\u001b[0m\u001b[1;36m)\u001b[0m                                                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mdb.create_table\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"my_table\"\u001b[0m\u001b[36m, \u001b[0m\u001b[33mdata\u001b[0m\u001b[36m=\u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m{\u001b[0m\u001b[32m\"vector\"\u001b[0m\u001b[36m: \u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m1.1\u001b[0m\u001b[36m, \u001b[0m\u001b[1;36m1.2\u001b[0m\u001b[1;36m]\u001b[0m\u001b[36m, \u001b[0m\u001b[32m\"b\"\u001b[0m\u001b[36m: \u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;36m}\u001b[0m\u001b[36m,\u001b[0m                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[33m...\u001b[0m\u001b[36m                                   \u001b[0m\u001b[1;36m{\u001b[0m\u001b[32m\"vector\"\u001b[0m\u001b[36m: \u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m0.5\u001b[0m\u001b[36m, \u001b[0m\u001b[1;36m1.3\u001b[0m\u001b[1;36m]\u001b[0m\u001b[36m, \u001b[0m\u001b[32m\"b\"\u001b[0m\u001b[36m: \u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;36m}\u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m                                   \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35mLanceTable\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mconnection\u001b[0m\u001b[36m=\u001b[0m\u001b[33m...\u001b[0m\u001b[36m, \u001b[0m\u001b[33mname\u001b[0m\u001b[36m=\u001b[0m\u001b[32m\"my_table\"\u001b[0m\u001b[1;36m)\u001b[0m                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mdb.create_table\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"another_table\"\u001b[0m\u001b[36m, \u001b[0m\u001b[33mdata\u001b[0m\u001b[36m=\u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m{\u001b[0m\u001b[32m\"vector\"\u001b[0m\u001b[36m: \u001b[0m\u001b[1;36m[\u001b[0m\u001b[1;36m0.4\u001b[0m\u001b[36m, \u001b[0m\u001b[1;36m0.4\u001b[0m\u001b[1;36m]\u001b[0m\u001b[36m, \u001b[0m\u001b[32m\"b\"\u001b[0m\u001b[36m: \u001b[0m\u001b[1;36m6\u001b[0m\u001b[1;36m}\u001b[0m\u001b[1;36m]\u001b[0m\u001b[1;36m)\u001b[0m                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35mLanceTable\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mconnection\u001b[0m\u001b[36m=\u001b[0m\u001b[33m...\u001b[0m\u001b[36m, \u001b[0m\u001b[33mname\u001b[0m\u001b[36m=\u001b[0m\u001b[32m\"another_table\"\u001b[0m\u001b[1;36m)\u001b[0m                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35msorted\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;35mdb.table_names\u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m)\u001b[0m\u001b[1;36m)\u001b[0m                                                                             \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m[\u001b[0m\u001b[32m'another_table'\u001b[0m\u001b[36m, \u001b[0m\u001b[32m'my_table'\u001b[0m\u001b[1;36m]\u001b[0m                                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mlen\u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mdb\u001b[0m\u001b[1;36m)\u001b[0m                                                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m2\u001b[0m                                                                                                        \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> db\u001b[0m\u001b[1;36m[\u001b[0m\u001b[32m\"my_table\"\u001b[0m\u001b[1;36m]\u001b[0m                                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35mLanceTable\u001b[0m\u001b[1;36m(\u001b[0m\u001b[33mconnection\u001b[0m\u001b[36m=\u001b[0m\u001b[33m...\u001b[0m\u001b[36m, \u001b[0m\u001b[33mname\u001b[0m\u001b[36m=\u001b[0m\u001b[32m\"my_table\"\u001b[0m\u001b[1;36m)\u001b[0m                                                              \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[32m\"my_table\"\u001b[0m\u001b[36m in db\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;92mTrue\u001b[0m                                                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mdb.drop_table\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"my_table\"\u001b[0m\u001b[1;36m)\u001b[0m                                                                            \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m>>> \u001b[0m\u001b[1;35mdb.drop_table\u001b[0m\u001b[1;36m(\u001b[0m\u001b[32m\"another_table\"\u001b[0m\u001b[1;36m)\u001b[0m                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m│\u001b[0m \u001b[1;35mLanceDBConnection\u001b[0m\u001b[1m(\u001b[0m\u001b[35m/home/msivanes/Documents/0Inbox/bclavie-beyond-basics-for-retrieval/\u001b[0m\u001b[95m.my_db\u001b[0m\u001b[1m)\u001b[0m        \u001b[32m│\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[32m╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;33mread_consistency_interval\u001b[0m = \u001b[3;35mNone\u001b[0m                                                                         \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                       \u001b[3;33muri\u001b[0m = \u001b[32m'/home/msivanes/Documents/0Inbox/bclavie-beyond-basics-for-retrieval/.my_db'\u001b[0m \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(db, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "tbl = db.create_table('my_table', schema=Document)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanceTable(connection=LanceDBConnection(/home/msivanes/Documents/0Inbox/bclavie-beyond-basics-for-retrieval/.my_db), name=\"my_table\")"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tbl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "tbl.add(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install tantivy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "tbl.create_fts_index('text') # Create a full-text search index on the 'text' field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What is Chihiro's new name given to her by the witch?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rinspect(tbl.search, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = (tbl.search(query, query_type='hybrid') # Uses LinearCombination by default with 0.7 for vector and 0.3 for full text search\n",
    ".where(\"category = 'film'\", prefilter=True)\n",
    ".limit(10)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_vec = (tbl.search(query, query_type='vector')\n",
    ".where(\"category = 'film'\", prefilter=True)\n",
    ".limit(10)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_fts = (tbl.search(query, query_type='fts')# Use the full-text search index\n",
    ".where(\"category = 'film'\", prefilter=True)\n",
    ".limit(10)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>vector</th>\n",
       "      <th>_distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Plot\\nTen-year-old Chihiro Ogino and her paren...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02793182, 0.019138196, -0.037934795, 0.035...</td>\n",
       "      <td>0.709315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Themes\\nSupernaturalism\\nThe major themes of S...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.012639961, -0.012689253, -0.060540363, 0.0...</td>\n",
       "      <td>0.809187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Stage \"Spirited Away\" (Chihiro role: Kanna Has...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.039504535, -0.040483247, 0.06785917, -0.04...</td>\n",
       "      <td>0.814720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Traditional Japanese culture\\nSpirited Away co...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.005438652, 0.051189482, 0.0004926561, -0.0...</td>\n",
       "      <td>0.847153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Stage adaptation\\nA stage adaptation of Spirit...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.055777222, -0.054559223, 0.059581153, -0.0...</td>\n",
       "      <td>0.864792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fantasy\\nThe film has been compared to Lewis C...</td>\n",
       "      <td>film</td>\n",
       "      <td>[0.026491445, 0.0057646018, 0.008504444, 0.033...</td>\n",
       "      <td>0.881455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Western consumerism\\nSimilar to the Japanese c...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.0036551235, 0.06056097, 0.002257561, 0.017...</td>\n",
       "      <td>0.894794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Environmentalism\\nCommentators have often refe...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02491368, -0.0074914335, -0.01859345, 0.03...</td>\n",
       "      <td>0.896202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The film's setting encapsulates Miyazaki's com...</td>\n",
       "      <td>film</td>\n",
       "      <td>[0.030683458, 0.006154182, -0.0024028437, 0.02...</td>\n",
       "      <td>0.939567</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Miyazaki had wanted to produce a new film for ...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.049608227, -0.0050955387, -0.027503341, 0....</td>\n",
       "      <td>0.944041</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text category  \\\n",
       "0  Plot\\nTen-year-old Chihiro Ogino and her paren...     film   \n",
       "1  Themes\\nSupernaturalism\\nThe major themes of S...     film   \n",
       "2  Stage \"Spirited Away\" (Chihiro role: Kanna Has...     film   \n",
       "3  Traditional Japanese culture\\nSpirited Away co...     film   \n",
       "4  Stage adaptation\\nA stage adaptation of Spirit...     film   \n",
       "5  Fantasy\\nThe film has been compared to Lewis C...     film   \n",
       "6  Western consumerism\\nSimilar to the Japanese c...     film   \n",
       "7  Environmentalism\\nCommentators have often refe...     film   \n",
       "8  The film's setting encapsulates Miyazaki's com...     film   \n",
       "9  Miyazaki had wanted to produce a new film for ...     film   \n",
       "\n",
       "                                              vector  _distance  \n",
       "0  [-0.02793182, 0.019138196, -0.037934795, 0.035...   0.709315  \n",
       "1  [-0.012639961, -0.012689253, -0.060540363, 0.0...   0.809187  \n",
       "2  [-0.039504535, -0.040483247, 0.06785917, -0.04...   0.814720  \n",
       "3  [-0.005438652, 0.051189482, 0.0004926561, -0.0...   0.847153  \n",
       "4  [-0.055777222, -0.054559223, 0.059581153, -0.0...   0.864792  \n",
       "5  [0.026491445, 0.0057646018, 0.008504444, 0.033...   0.881455  \n",
       "6  [-0.0036551235, 0.06056097, 0.002257561, 0.017...   0.894794  \n",
       "7  [-0.02491368, -0.0074914335, -0.01859345, 0.03...   0.896202  \n",
       "8  [0.030683458, 0.006154182, -0.0024028437, 0.02...   0.939567  \n",
       "9  [-0.049608227, -0.0050955387, -0.027503341, 0....   0.944041  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_vec.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>vector</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Plot\\nTen-year-old Chihiro Ogino and her paren...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02793182, 0.019138196, -0.037934795, 0.035...</td>\n",
       "      <td>17.426216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Spirited Away (Japanese: 千と千尋の神隠し, Hepburn: Se...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02796123, -0.027909389, -0.0047542905, 0.0...</td>\n",
       "      <td>9.585370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fantasy\\nThe film has been compared to Lewis C...</td>\n",
       "      <td>film</td>\n",
       "      <td>[0.026491445, 0.0057646018, 0.008504444, 0.033...</td>\n",
       "      <td>8.506400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Western consumerism\\nSimilar to the Japanese c...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.0036551235, 0.06056097, 0.002257561, 0.017...</td>\n",
       "      <td>8.375582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Music\\nThe film score of Spirited Away was com...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.049314238, -0.015812675, 0.0023815727, -0....</td>\n",
       "      <td>6.391144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>English dub\\nJohn Lasseter, Pixar animator and...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.0728673, -0.048440374, 0.01923553, 0.01419...</td>\n",
       "      <td>6.218455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>The film's setting encapsulates Miyazaki's com...</td>\n",
       "      <td>film</td>\n",
       "      <td>[0.030683458, 0.006154182, -0.0024028437, 0.02...</td>\n",
       "      <td>6.092483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Reception\\nCritical response\\nSpirited Away re...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.042472918, -0.026727537, -0.016023315, 0.0...</td>\n",
       "      <td>5.953347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Themes\\nSupernaturalism\\nThe major themes of S...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.012639961, -0.012689253, -0.060540363, 0.0...</td>\n",
       "      <td>5.949172</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text category  \\\n",
       "0  Plot\\nTen-year-old Chihiro Ogino and her paren...     film   \n",
       "1  Spirited Away (Japanese: 千と千尋の神隠し, Hepburn: Se...     film   \n",
       "2  Fantasy\\nThe film has been compared to Lewis C...     film   \n",
       "3  Western consumerism\\nSimilar to the Japanese c...     film   \n",
       "4  Music\\nThe film score of Spirited Away was com...     film   \n",
       "5  English dub\\nJohn Lasseter, Pixar animator and...     film   \n",
       "6  The film's setting encapsulates Miyazaki's com...     film   \n",
       "7  Reception\\nCritical response\\nSpirited Away re...     film   \n",
       "8  Themes\\nSupernaturalism\\nThe major themes of S...     film   \n",
       "\n",
       "                                              vector      score  \n",
       "0  [-0.02793182, 0.019138196, -0.037934795, 0.035...  17.426216  \n",
       "1  [-0.02796123, -0.027909389, -0.0047542905, 0.0...   9.585370  \n",
       "2  [0.026491445, 0.0057646018, 0.008504444, 0.033...   8.506400  \n",
       "3  [-0.0036551235, 0.06056097, 0.002257561, 0.017...   8.375582  \n",
       "4  [-0.049314238, -0.015812675, 0.0023815727, -0....   6.391144  \n",
       "5  [-0.0728673, -0.048440374, 0.01923553, 0.01419...   6.218455  \n",
       "6  [0.030683458, 0.006154182, -0.0024028437, 0.02...   6.092483  \n",
       "7  [-0.042472918, -0.026727537, -0.016023315, 0.0...   5.953347  \n",
       "8  [-0.012639961, -0.012689253, -0.060540363, 0.0...   5.949172  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_fts.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>vector</th>\n",
       "      <th>_relevance_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Plot\\nTen-year-old Chihiro Ogino and her paren...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02793182, 0.019138196, -0.037934795, 0.035...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Themes\\nSupernaturalism\\nThe major themes of S...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.012639961, -0.012689253, -0.060540363, 0.0...</td>\n",
       "      <td>0.402163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Stage \"Spirited Away\" (Chihiro role: Kanna Has...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.039504535, -0.040483247, 0.06785917, -0.04...</td>\n",
       "      <td>0.385660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Traditional Japanese culture\\nSpirited Away co...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.005438652, 0.051189482, 0.0004926561, -0.0...</td>\n",
       "      <td>0.288938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fantasy\\nThe film has been compared to Lewis C...</td>\n",
       "      <td>film</td>\n",
       "      <td>[0.026491445, 0.0057646018, 0.008504444, 0.033...</td>\n",
       "      <td>0.253488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Stage adaptation\\nA stage adaptation of Spirit...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.055777222, -0.054559223, 0.059581153, -0.0...</td>\n",
       "      <td>0.236335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Spirited Away (Japanese: 千と千尋の神隠し, Hepburn: Se...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02796123, -0.027909389, -0.0047542905, 0.0...</td>\n",
       "      <td>0.221776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Western consumerism\\nSimilar to the Japanese c...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.0036551235, 0.06056097, 0.002257561, 0.017...</td>\n",
       "      <td>0.210289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Environmentalism\\nCommentators have often refe...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02491368, -0.0074914335, -0.01859345, 0.03...</td>\n",
       "      <td>0.142666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Music\\nThe film score of Spirited Away was com...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.049314238, -0.015812675, 0.0023815727, -0....</td>\n",
       "      <td>0.026956</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text category  \\\n",
       "0  Plot\\nTen-year-old Chihiro Ogino and her paren...     film   \n",
       "1  Themes\\nSupernaturalism\\nThe major themes of S...     film   \n",
       "2  Stage \"Spirited Away\" (Chihiro role: Kanna Has...     film   \n",
       "3  Traditional Japanese culture\\nSpirited Away co...     film   \n",
       "4  Fantasy\\nThe film has been compared to Lewis C...     film   \n",
       "5  Stage adaptation\\nA stage adaptation of Spirit...     film   \n",
       "6  Spirited Away (Japanese: 千と千尋の神隠し, Hepburn: Se...     film   \n",
       "7  Western consumerism\\nSimilar to the Japanese c...     film   \n",
       "8  Environmentalism\\nCommentators have often refe...     film   \n",
       "9  Music\\nThe film score of Spirited Away was com...     film   \n",
       "\n",
       "                                              vector  _relevance_score  \n",
       "0  [-0.02793182, 0.019138196, -0.037934795, 0.035...          1.000000  \n",
       "1  [-0.012639961, -0.012689253, -0.060540363, 0.0...          0.402163  \n",
       "2  [-0.039504535, -0.040483247, 0.06785917, -0.04...          0.385660  \n",
       "3  [-0.005438652, 0.051189482, 0.0004926561, -0.0...          0.288938  \n",
       "4  [0.026491445, 0.0057646018, 0.008504444, 0.033...          0.253488  \n",
       "5  [-0.055777222, -0.054559223, 0.059581153, -0.0...          0.236335  \n",
       "6  [-0.02796123, -0.027909389, -0.0047542905, 0.0...          0.221776  \n",
       "7  [-0.0036551235, 0.06056097, 0.002257561, 0.017...          0.210289  \n",
       "8  [-0.02491368, -0.0074914335, -0.01859345, 0.03...          0.142666  \n",
       "9  [-0.049314238, -0.015812675, 0.0023815727, -0....          0.026956  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install cohere"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lancedb.rerankers import CohereReranker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">class</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'lancedb.rerankers.cohere.CohereReranker'</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">&gt;</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff; font-style: italic\">class </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">CohereReranker</span><span style=\"font-weight: bold\">(</span>model_name: str = <span style=\"color: #008000; text-decoration-color: #008000\">'rerank-english-v2.0'</span>, column: str = <span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>, top_n: Optional<span style=\"font-weight: bold\">[</span>int<span style=\"font-weight: bold\">]</span> =      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>, <span style=\"color: #808000; text-decoration-color: #808000\">return_score</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'relevance'</span>, api_key: Optional<span style=\"font-weight: bold\">[</span>str<span style=\"font-weight: bold\">]</span> = <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span><span style=\"font-weight: bold\">)</span>:                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Reranks the results using the Cohere Rerank API.</span>                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff; text-decoration: underline\">https://docs.cohere.com/docs/rerank-guide</span>                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">Parameters</span>                                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">----------</span>                                                                                                      <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">model_name : str, default </span><span style=\"color: #008000; text-decoration-color: #008000\">\"rerank-english-v2.0\"</span>                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    The name of the cross encoder model to use. Available cohere models are:</span>                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    - rerank-english-v2.</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>                                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    - rerank-multilingual-v2.</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">column : str, default </span><span style=\"color: #008000; text-decoration-color: #008000\">\"text\"</span>                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    The name of the column to use as input to the cross encoder model.</span>                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">top_n : str, default </span><span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>                                                                                       <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">    The number of results to return. If </span><span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span><span style=\"color: #008080; text-decoration-color: #008080\">, will return all results.</span>                                          <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span><span style=\"font-style: italic\"> attribute(s) not shown.</span> Run <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">inspect</span><span style=\"font-weight: bold\">(</span>inspect<span style=\"font-weight: bold\">)</span> for options.                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34m<\u001b[0m\u001b[1;95mclass\u001b[0m\u001b[39m \u001b[0m\u001b[32m'lancedb.rerankers.cohere.CohereReranker'\u001b[0m\u001b[1;34m>\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;96mclass \u001b[0m\u001b[1;31mCohereReranker\u001b[0m\u001b[1m(\u001b[0mmodel_name: str = \u001b[32m'rerank-english-v2.0'\u001b[0m, column: str = \u001b[32m'text'\u001b[0m, top_n: Optional\u001b[1m[\u001b[0mint\u001b[1m]\u001b[0m =      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[3;35mNone\u001b[0m, \u001b[33mreturn_score\u001b[0m=\u001b[32m'relevance'\u001b[0m, api_key: Optional\u001b[1m[\u001b[0mstr\u001b[1m]\u001b[0m = \u001b[3;35mNone\u001b[0m\u001b[1m)\u001b[0m:                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mReranks the results using the Cohere Rerank API.\u001b[0m                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[4;94mhttps://docs.cohere.com/docs/rerank-guide\u001b[0m                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mParameters\u001b[0m                                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m----------\u001b[0m                                                                                                      \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mmodel_name : str, default \u001b[0m\u001b[32m\"rerank-english-v2.0\"\u001b[0m                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    The name of the cross encoder model to use. Available cohere models are:\u001b[0m                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    - rerank-english-v2.\u001b[0m\u001b[1;36m0\u001b[0m                                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    - rerank-multilingual-v2.\u001b[0m\u001b[1;36m0\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mcolumn : str, default \u001b[0m\u001b[32m\"text\"\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    The name of the column to use as input to the cross encoder model.\u001b[0m                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36mtop_n : str, default \u001b[0m\u001b[3;35mNone\u001b[0m                                                                                       \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m    The number of results to return. If \u001b[0m\u001b[3;35mNone\u001b[0m\u001b[36m, will return all results.\u001b[0m                                          \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m32\u001b[0m\u001b[3m attribute(s) not shown.\u001b[0m Run \u001b[1;35minspect\u001b[0m\u001b[1m(\u001b[0minspect\u001b[1m)\u001b[0m for options.                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rinspect(CohereReranker, help=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os; assert os.environ['COHERE_API_KEY']\n",
    "reranker = CohereReranker(api_key=os.environ['COHERE_API_KEY'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_reranker = (tbl.search(query, query_type='hybrid')\n",
    ".where(\"category = 'film'\", prefilter=True)\n",
    ".limit(10)\n",
    ".rerank(reranker=reranker)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/msivanes/miniconda3/envs/rag/lib/python3.11/site-packages/lancedb/rerankers/cohere.py:88: FutureWarning: promote has been superseded by mode='default'.\n",
      "  combined_results = self.merge_results(vector_results, fts_results)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>vector</th>\n",
       "      <th>_relevance_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Plot\\nTen-year-old Chihiro Ogino and her paren...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02793182, 0.019138196, -0.037934795, 0.035...</td>\n",
       "      <td>0.993661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Spirited Away (Japanese: 千と千尋の神隠し, Hepburn: Se...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02796123, -0.027909389, -0.0047542905, 0.0...</td>\n",
       "      <td>0.905322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fantasy\\nThe film has been compared to Lewis C...</td>\n",
       "      <td>film</td>\n",
       "      <td>[0.026491445, 0.0057646018, 0.008504444, 0.033...</td>\n",
       "      <td>0.831964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Western consumerism\\nSimilar to the Japanese c...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.0036551235, 0.06056097, 0.002257561, 0.017...</td>\n",
       "      <td>0.716417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Stage \"Spirited Away\" (Chihiro role: Kanna Has...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.039504535, -0.040483247, 0.06785917, -0.04...</td>\n",
       "      <td>0.570391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The film's setting encapsulates Miyazaki's com...</td>\n",
       "      <td>film</td>\n",
       "      <td>[0.030683458, 0.006154182, -0.0024028437, 0.02...</td>\n",
       "      <td>0.340376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Environmentalism\\nCommentators have often refe...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.02491368, -0.0074914335, -0.01859345, 0.03...</td>\n",
       "      <td>0.332961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Traditional Japanese culture\\nSpirited Away co...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.005438652, 0.051189482, 0.0004926561, -0.0...</td>\n",
       "      <td>0.290981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Themes\\nSupernaturalism\\nThe major themes of S...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.012639961, -0.012689253, -0.060540363, 0.0...</td>\n",
       "      <td>0.207215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Stage adaptation\\nA stage adaptation of Spirit...</td>\n",
       "      <td>film</td>\n",
       "      <td>[-0.055777222, -0.054559223, 0.059581153, -0.0...</td>\n",
       "      <td>0.114171</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text category  \\\n",
       "0  Plot\\nTen-year-old Chihiro Ogino and her paren...     film   \n",
       "1  Spirited Away (Japanese: 千と千尋の神隠し, Hepburn: Se...     film   \n",
       "2  Fantasy\\nThe film has been compared to Lewis C...     film   \n",
       "3  Western consumerism\\nSimilar to the Japanese c...     film   \n",
       "4  Stage \"Spirited Away\" (Chihiro role: Kanna Has...     film   \n",
       "5  The film's setting encapsulates Miyazaki's com...     film   \n",
       "6  Environmentalism\\nCommentators have often refe...     film   \n",
       "7  Traditional Japanese culture\\nSpirited Away co...     film   \n",
       "8  Themes\\nSupernaturalism\\nThe major themes of S...     film   \n",
       "9  Stage adaptation\\nA stage adaptation of Spirit...     film   \n",
       "\n",
       "                                              vector  _relevance_score  \n",
       "0  [-0.02793182, 0.019138196, -0.037934795, 0.035...          0.993661  \n",
       "1  [-0.02796123, -0.027909389, -0.0047542905, 0.0...          0.905322  \n",
       "2  [0.026491445, 0.0057646018, 0.008504444, 0.033...          0.831964  \n",
       "3  [-0.0036551235, 0.06056097, 0.002257561, 0.017...          0.716417  \n",
       "4  [-0.039504535, -0.040483247, 0.06785917, -0.04...          0.570391  \n",
       "5  [0.030683458, 0.006154182, -0.0024028437, 0.02...          0.340376  \n",
       "6  [-0.02491368, -0.0074914335, -0.01859345, 0.03...          0.332961  \n",
       "7  [-0.005438652, 0.051189482, 0.0004926561, -0.0...          0.290981  \n",
       "8  [-0.012639961, -0.012689253, -0.060540363, 0.0...          0.207215  \n",
       "9  [-0.055777222, -0.054559223, 0.059581153, -0.0...          0.114171  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_reranker.to_pandas()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hayward",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}