ericstj

Design: Post-processing (special-token templates) for Microsoft.ML.Tokenizers

Status: prototype (PR #7625). Shareable design asset for review.

Problem

SentencePieceTokenizer.CreateFromTokenizerJson loads Hugging Face JSON-only Unigram tokenizers. HF tokenizer.json declares how special tokens wrap an encoded sequence in a post_processor block. The library has no post-processing abstraction today; SentencePiece only supports a single BOS + single EOS via the addBeginningOfSentence / addEndOfSentence switches. That cannot represent common real-world templates.

Data (sampled popular Unigram models on HF, n=57 resolved)

ericstj

using System;

namespace appsettings
{
    class Program
    {
        static void Main(string[] args)
        {
            System.Configuration.ConfigurationManager.AppSettings["hello"] = "world";
            Console.WriteLine("Hello World!");

ericstj

##dotnet/CoreFx ###Libraries in NETStandard

• ref
  • Default targetgroup should be NETCoreApp build
  • P2P references to other reference assembly CSProjs.
  • System.Runtime core assembly.
  • Cross-compiles for concrete frameworks (if different)
    • EG: exposes types/members not in NETStandard on NETCoreApp, but not on UWP
• src
• Default targetgroup should be NETCoreApp build

ericstj

API Difference C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETFramework\v4.6.1\System.Net.Http.dll vs C:\Users\ericstj.nuget\packages\System.Net.Http\4.1.0\ref\netstandard1.1\System.Net.Http.dll

API listing follows standard diff formatting. Lines preceded by a '+' are additions and a '-' indicates removal.

System.Net.Http

 namespace System.Net.Http {
     public class ByteArrayContent : HttpContent {

ericstj

Resolving binaries from NuGet packages

NuGet is handy for sharing code but sometimes you need to deal with raw binaries. Picking those directly out of NuGet packages can be challenging since NuGet has many complex/inter-related conventions for determining which assets are applicable.

This document will describe a technique for resolving binaries using NuGet's asset selection algorithm via DotNET CLI.

Prerequisites

ericstj

{
  "buildOptions": {
    "preserveCompilationContext": true
  },
  "dependencies": {
    "Microsoft.NETCore.App": {
      "type": "platform",
      "version": "1.0.0-rc3-004308"
    },
    "Microsoft.CSharp": "4.0.1",

ericstj

###core-setup

Package Id	Version
Microsoft.NETCore.App	1.0.0
Microsoft.NETCore	5.0.1
Microsoft.NETCore.UniversalWindowsPlatform	5.2.0
Microsoft.NETCore.DotNetHost	1.0.1
Microsoft.NETCore.DotNetHostPolicy	1.0.1
Microsoft.NETCore.DotNetHostResolver	1.0.1
runtime.debian.8-x64.Microsoft.NETCore.DotNetHost	1.0.1

ericstj

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <PropertyGroup>
        <OutputPath>DTAR_08E86330_4835_4B5C_9E5A_61F37AE1A077_DTAR</OutputPath>
        <TargetFrameworkIdentifier>.NETCore</TargetFrameworkIdentifier>
        <TargetFrameworkVersion>v5.0</TargetFrameworkVersion>
        <TargetFrameworkProfile></TargetFrameworkProfile>
        <TargetPlatformIdentifier>UAP</TargetPlatformIdentifier>
        <TargetPlatformVersion>10.0</TargetPlatformVersion>
        <FrameworkRegistryBase></FrameworkRegistryBase>

ericstj

Error : Multiple projects built twice with the same target path C:\src\corefx\bin\ref\System.Reflection.TypeExtensions\4.1.0.0\System.Reflection.TypeExtensions.dll.
C:\src\corefx\build.proj -->
C:\src\corefx\src\dirs.proj -->
  Global Properties:
    DefaultBuildAllTarget = BuildAndTest
    Dummy = src\dirs.proj
    FilterToOSGroup = Windows_NT
    BuildAllProjects = true
C:\src\corefx\src\tests.builds -->
  Global Properties: