richdougherty

llama-cpp-python LoRA hot-swapping demo

Introduction

This document explains how to do LoRAs hot-swapping via the llama.cpp Python wrapper, llama-cpp-python.

This provides extra docs for my pull request: abetlen/llama-cpp-python#1817

For now, the main llama-cpp-python main branch doesn't have LoRA hot-swapping support merged, so this guide uses the code for the branch to test out LoRA support.

richdougherty

// ==UserScript==
// @name         Connections Categoriser
// @description  Adds a category function to the NYT Connections game
// @match        https://www.nytimes.com/games/connections
// @namespace    http://tampermonkey.net/
// @version      0.1.0
// @grant        none
// ==/UserScript==

// MIT License

richdougherty

#!/bin/bash
# Copyright 2023 Rich Dougherty

# which-hunt - an alternative to 'which' that traces symlinks and shims

# - Examples -
#
# $ which java
# /usr/bin/java
#

richdougherty

Tested with https://github.com/rbsec/sslscan v1.11.11


===== Java 1.6.0_45 =====

Testing SSL server localhost on port 9443 using SNI name localhost

  TLS Fallback SCSV:
Server does not support TLS Fallback SCSV

richdougherty

trait RequestFactory {
  def createRequestHeader(
    connection: RemoteConnection,
    method: String,
    target: RequestTarget,
    version: String,
    headers: Headers,
    attrs: TypedMap): RequestHeader
}

richdougherty

trait GreeterService {
  ...  
}

object GreeterService extends GreeterServiceCompanion[GreeterService] {
  override val name = "helloworld.GreeterService"
  ...
}

class GreeterServiceClient(channel: Channel, options: CallOptions)(implicit mat: Materializer, ex: ExecutionContext) extends GreeterService {

richdougherty

trait Handler

trait HandlerOp[+A]
case class HandlerVal[+A](value: A) extends HandlerOp[A]
case class HandlerThunk[+A](thunk: () => A) extends HandlerOp[A]
case class HandlerFuture[+A](future: Future[A]) extends HandlerOp[A]
case class HandlerEither[+A,B<:A,C<:A](either: Either[B,C]) extends HandlerOp[A]

object Handler {
  trait RequestHeader

richdougherty

/*
 * Copyright (C) 2009-2017 Lightbend Inc. <https://www.lightbend.com>
 */
package play.it.http

import ch.qos.logback.classic.Logger
import ch.qos.logback.classic.spi.ILoggingEvent
import ch.qos.logback.core.AppenderBase
import org.slf4j.LoggerFactory
import org.specs2.execute.AsResult

richdougherty

Use CS to shut down Server and Application in Play

Changes to ApplicationLifecycle usage

• Application components systems can continue to use ApplicationLifecycle
• Can now also use CoordinatedShutdown if want richer API
  • Suggest that we don't change ApplicationLifecycle API to expose phases (yet)
• Application can now create its own ApplicationLifecycle from the ActorSystem
  • Calls ApplicationLifecycle.stop() during a phase of CS
• ApplicationLifecycle manages ordering (as suggested by Patrik)

richdougherty

// Change DefaultHttpRequestHandler to take a pipeline

class DefaultHttpRequestHandler(
    pipeline: Seq[Handler],
    router: Router,
    errorHandler: HttpErrorHandler,
    configuration: HttpConfiguration,
    filters: Seq[EssentialFilter]) extends HttpRequestHandler {
  ...
}