Linter for Factor USING: ... ; forms

lint-using

! Copyright (C) Charles Alston
! See http://factorcode.org/license.txt for BSD license
USING: accessors arrays fry io io.backend io.directories.search
io.encodings.utf8 io.files io.pathnames kernel parser regexp
sequences tools.crossref vocabs vocabs.refresh wrap.strings ;
IN: lint-using

! gist title: Linter for Factor USING: ... ; forms
! gist URL:
! https://gist.github.com/cwalston/f6da9cca0105f5d67483935380001d84
 
! Remove superfluous vocabs declared in USING: ... ; forms.
! Test by adding & saving unnecessary vocab declarations to a working
! vocab USING: ... ; form & calling `lint-using` on the vocab.
! (As usual, place in an eponymously named folder in your work directory,
!  & back up your stuff, to test)

! matches a well-formed sub-sequence of "USING: ... ;" in a source file.
CONSTANT: USING-list-regexp  R/ USING:(\s+[^;]+)+\s+;/ ! Thanks, John
 
: first-using-list ( string -- slice )
    USING-list-regexp first-match ;

: write-USING ( vocab-name -- string )
  vocab-uses           ! ( -- seq )
  "USING:" prefix ";" suffix " " join    
  64 wrap-string ;     ! ( -- string  )

: work-vocabs>source-path ( vocab-name -- abspath )
    [ "resource:work" normalize-path 1array ] dip     ! ( -- dir-path name )
    file-name ".factor" append
    '[ file-name _ = ]
    find-file-in-directories ;

: replace-USING ( string vocab-name -- string' )
    work-vocabs>source-path utf8 file-contents
    first-using-list  ! ( -- string slice )
    [ from>> ] [ to>> ] [ seq>> ] tri replace-slice ; 

! turn on auto-use to fill in minimal list
: reload-linted ( string vocab-name -- )  
    work-vocabs>source-path utf8 set-file-contents   ! ( -- )
    auto-use refresh-all ;

! vocab-name can be a plain vocab name, e.g., "3way-search",
! a filename w/ extension, e.g., "3way-search.factor",
! a nested form, e.g., "resource:work/search-herbal/3way-search/3way-search.factor",
! or a fully qualified work directory path. 
! Searches only source files in "resource:work" directory.

! - Listener lint utility for USING: ... ; forms, in loaded vocabs.
! - Press F2 after executing `lint-using` to display
!   linted USING: ... ; form for copy/paste/save to target source file.

! example tests on my system:
! execute `<vocab-name> lint-using` in Listener, e.g.,
! "herbal-article-element" lint-using  ! Press F2 after this, 
! "3way-search" lint-using             ! to invoke refresh/restarts
! "open-query-browser" lint-using
: lint-using ( vocab-name -- )
    [ write-USING ]          ! ( -- string )
    [ replace-USING ]        ! ( -- string' )
    [ reload-linted ] tri ;  ! ( -- )

! convenience word & alias to distinguish loaded/not-loaded vocabs
: lint-if-loaded ( vocab-name -- )
    dup lookup-vocab   ! ( -- vocab-name T{ vocab }/f )
    [ lint-using ] 
    [ "``" "'' " surround "is not loaded." append print ] if ;

ALIAS: lil lint-if-loaded

! Lint workflow looks like this (tl;dr):
! (1) open <vocab-name> in your editor,
! (2) copy <vocab-name> at the IN: line (or wherever convenient, per your editor),
! (3) in Listener, enter: "<vocab-name>" lint-if-loaded (double-quoted)
!    (easier, enter "<vocab-name>" lil); press <ret>,
! (4) you'll see either the msg: ``<vocab-name>'' is not loaded. (so skip this vocab), 
!     or auto-use is on; if on,
! (5) press F2 for restart/refresh, & copy the presented USING: ... ; form (now linted), 
!     paste over the form in your editor & save the file (close it if you want).
!    (Note: when you switch back to your editor, the USING form there will shrink, 
!     typically; just paste the newly copied linted form over what is there, & save.
!     - If no restarts are triggered & no USING revisions appear, 
!       return to editor & save vocab with USING form as is or has changed.)
! (6) in Listener, press F2 to reload the saved vocab file; turn off auto-use.
! Done; the vocab is saved & reloaded, with its USING: ... ; form linted.
! Rinse & repeat to lint USING: ... ; forms in other vocabs.
! -----------------------------------------------------------------------------------
! to do: work out routines for these other syntax words --
! USE: vocab
! UNUSE: vocab
! FROM: vocab => words ... ;
! EXCLUDE: vocab => word ... ;
! QUALIFIED: vocab
! QUALIFIED-WITH: vocab name

Well, it's much farther than I've gotten (I haven't put my ideas into code yet) but here, I really don't think that [ see ... ] with-file-writerapproach is very useful. Because, it is more trouble than it is worth and you should use see* and the words it calls instead. Plus, see factor/factor#1561 (long words can't be helped; see will fail to round-trip for long definitions)

Valid instances of the word class have a dependencies property, you can see it like \ zero? "dependencies" word-prop or \ zero? props>> "dependencies" of. All you have to do then is write vocabulary>> to get the string name of the vocabulary that contains each word, set-like to make it a unique list, and compare this against the USING: and other declarations.

This would be a lot easier with the refactoring abilities of the new parser we'd like to get finished for 0.99, but... if you want to use the see machinery, another way is to assemble a manifest for all the vocab words then write it out as a single using list:

:: vocab-using ( vocab -- using )
    vocab vocab-words <manifest> [
        [
           {
              [ synopsis* ]
              [ definition pprint-elements ]
              [ definer nip [ pprint-word ] when* ]
              [ declarations. ]
           } cleave
        ] t make-pprint nip
            [ search-vocabs>> '[ _ over adjoin-all ] change-search-vocabs ]
            [ qualified-vocabs>> '[ _ over adjoin-all ] change-qualified-vocabs ] bi 
    ] reduce search-vocabs>> vocab vocab-name ".private" append over delete natural-sort ;

You can see it works more or less (missing fry since that turns into a bunch of curry instructions) and gets locals wrong (including locals.backend instead of locals for a similar reason to fry).

For example on vocabs like calendars.holiday.

IN: scratchpad "calendar.holidays" vocab-using .
{
    "accessors"
    "assocs"
    "calendar"
    "combinators"
    "kernel"
    "locals.backend"
    "parser"
    "sequences"
    "words"
}

vs.

USING: accessors assocs calendar fry kernel locals parser
sequences vocabs words ;

Probably could fix that bug and then it would be pretty good, except for any top-level code that might have USE: declarations that aren't used by any word definitions, and it wouldn't get the FROM: forms correct for writing back, but the new parser fixes all that.

Maybe if you just built a help.lint.using warning system and then the user could manually confirm/fix, that would be a good first step?

Alternatively, we could preserve the manifest somewhere that was created as part of parsing the vocabulary and then you'd have the exact one the vocabulary was parsed with.