splitting text w/o spaces into words

find-words.lisp

(ql:quickload :cl-ppcre)
(ql:quickload :rutils)

(use-package :rutil)
(named-readtables:in-readtable rutils-readtable)

(defpar *orig*
  #/I joined Hacker News around 5 years ago. I used to wake up and do the grim commute each morning to London from my home and the only thing that made it vaguely ok was Hacker News. It was a great place to go and find interesting articles from genuinely passionate people. It also used to be a really safe place to launch a startup that you'd spent days, weeks, years on - your project. It was a place where you could launch your startup and know you'd get great constructive feedback. People may not necessarily like your site but they'd admire you for having the balls to launch it, for spending time developing something that you hoped could benefit people in some way. They'd want you to succeed and they'd try and help you succeed with feedback that would ultimately help you. Unfortunately, today's Hacker News audience is no longer the same. Today's Hacker News is a place where users want to snipe at other users and find negative aspects to anything thing submitted. No longer does someone say 'This and this I like but this needs work'. Oh no, now the response is 'Hate this, hate that, this is pointless.'. Hacker News now is about correcting grammar and points scoring. It is pointing out anything negative at all that anyone has done, has said. It is no longer a safe place. It has fast become an acidic forum.
I've launched 2 projects (11kclub and Favilous) on here over the last year - both got a similar response. There was nothing constructive, it was just sniping - they saw someone had put themselves up there and they just shot them down. It's a real shame. I hope one day the site returns with the kind of audience it once had. Until that happens, I won't be going on my favourite site anymore - the commute just got a whole lot longer.

For now I wish you all the best...

Thanks

Steve

It's a genuine problem and has been growing gradually worse for a while. I think the cause is simply growth. When a good community grows, it becomes worse in two ways: (a) more recent arrivals don't have as much of whatever quality distinguished the original members, and (b) the large size of the group makes people behave worse, because there is more anonymity in a larger group.
I've spent many hours over the past several years trying to understand and mitigate such problems. I've come up with a bunch of tweaks that worked, and I have hopes I'll be able to come up with more.
The idea I'm currently investigating, in case anyone is curious, is that votes rather than comments may be the easiest place to attack this problem. Although snarky comments themselves are the most obvious symptom, I suspect that voting is on average dumber than commenting, because it requires so much less work. So I'm going to try to see if it's possible to identify people who consistently upvote nasty comments and if so count their votes less./#)

(defpar *test* (fmt "~{~A~}" (ppcre:all-matches-as-strings "\\w+" *orig*)))
(defpar n (length *test*))  ; n = 2239

(defpar *dict* #{equalp})
(dolist (word (ppcre:all-matches-as-strings "\\w+" *orig*))
  (set# word *dict* t))

(hash-table-count *dict*)  ; m = 269

(defpar k (apply #'max (mapcar #'length (ht-keys *dict*))))  ; k = 13

;; complexity < (k + 1) * n ~ O(n)

(let ((reached (make-array n :initial-element nil)))
  (setf (elt reached 0) t)
  (block outer
    (do ((i 0 (1+ i))
         (c 0)
         (rez (make-array n :initial-element nil)))
        ((= i n))
      (when (elt reached i)
        (dotimes (j k)
          (let ((l (+ i j 1)))
            (when (and (<= l n) 
                       (not (elt reached l))
              (incf c)
              (let ((w (subseq *test* i l)))
                (when (get# w *dict*)
                  (push w (elt rez i))
                  (when (= l n)
                    (return-from outer
                      (values (apply #'strjoin " "
                                     (let ((prev 1)
                                           text)
                                       (dotimes (i n text)
                                         (if-it (and-it (elt rez (- n i 1))
                                                        (find-if #`(= prev (length %))
                                                                 it))
                                                (progn (push it text)
                                                       (setf prev 1))
                                                (incf prev)))))
                              c)))
                  (setf (elt reached l) t))))))))))

; Result: "I joined Hacker News around 5 years ago I used to wake up and do the grim commute each morning to London from my home and the only thing that made it vaguely ok was Hacker News It was a great place to go and find interesting articles from genuinely passionate people It also used to be a really safe place to launch a startup that you d spent days weeks years on your project It was a place where you could launch your startup and know you d get great constructive feedback People may not necessarily like your site but they d admire you for having the balls to launch it for spending time developing something that you hoped could benefit people in some way They d want you to succeed and they d try and help you succeed with feedback that would ultimately help you Unfortunately to days Hacker News audience is no longer the same To days Hacker News is a place where users want to snipe at other users and find negative aspects to anything thing submitted No longer does someone say This and this I like but this needs work Oh no now the response is Hate this hate that this is pointless Hacker News now is about correcting grammar and points scoring It is pointing out anything negative at all that anyone has done has said It is no longer a safe place It has fast become an acidic forum I ve launched 2 projects 11kclub and Favilous on here over the last year both got a similar response There was nothing constructive it was just sniping they saw someone had put themselves up there and they just shot them down It s a real shame I hope one day the site returns with the kind of audience it once had Until that happens I won t be going on my favourite site anymore the commute just got a whole lot longer For now I wish you all the best Thanks Steve It s a genuine problem and has been growing gradually worse for a while I think the cause is simply growth When a good community grows it becomes worse in two ways a more recent arrivals don t have as much of whatever quality distinguished the original members and b the large size of the group makes people behave worse because there is more anonymity in a larger group I ve spent many hours over the past several years trying to understand and mitigate such problems I ve come up with a bunch of tweaks that worked and I have hopes I ll be able to come up with more The idea I m currently investigating in case anyone is curious is that votes rather than comments may be the easiest place to attack this problem Although snarky comments themselves are the most obvious symptom I suspect that voting is on average dumber than commenting because it requires so much less work So I m going to try to see if it s possible to identify people who consistently upvote nasty comments and if so count their votes less"
; 11933