Skip to content

Instantly share code, notes, and snippets.

@MgaMPKAy
Created May 12, 2012 13:31
Show Gist options
  • Save MgaMPKAy/2666526 to your computer and use it in GitHub Desktop.
Save MgaMPKAy/2666526 to your computer and use it in GitHub Desktop.
download shanbay wordlist in json format
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import Data.Conduit
import Data.Conduit.Binary (sinkFile)
import Data.Conduit.Attoparsec (sinkParser)
import Data.Aeson.Parser (json)
import Control.Monad (when)
import qualified Data.ByteString.Lazy as L
import Control.Monad.IO.Class (liftIO)
main :: IO ()
main = do
runResourceT $ do
manager <- liftIO $ newManager def
shanbay manager 15574 2000 30
shanbay manager wordlistId total step =
shanbay' manager wordlistId total step 1
where
shanbay' manager wordlistId total step count
| count > total = return ()
| otherwise = do
let url = "http://www.shanbay.com/wordlist/vocabularies/?"
++ "id=" ++ show wordlistId
++ "&start=" ++ show count
++ "&offset=30"
req <- liftIO $ parseUrl url
res <- http req manager
-- j <- responseBody res $$ sinkParser json
-- liftIO $ print j
responseBody res $$ sinkFile $ "wordlist" ++ show count ++ ".txt"
shanbay' manager wordlistId total step (count + step)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment