Skip to content

Instantly share code, notes, and snippets.

@ririw

ririw/Compilation and running

Last active January 1, 2016 21:59
Show Gist options
  • Save ririw/8207250 to your computer and use it in GitHub Desktop.
Save ririw/8207250 to your computer and use it in GitHub Desktop.
Download ZIP
New version of reader code
Raw
Compilation and running
ghc -O2 -rtsopts -threaded -prof -fprof-auto -fforce-recomp reader.hs
time ./reader +RTS -K1G -sstderr -pa -A3M
Raw
Dataset sample
# started 2013-08-04T11:34:31Z
<http://dbpedia.org/resource/AccessibleComputing> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Computer_accessibility> .
<http://dbpedia.org/resource/AfghanistanGeography> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Geography_of_Afghanistan> .
<http://dbpedia.org/resource/AfghanistanHistory> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/History_of_Afghanistan> .
<http://dbpedia.org/resource/AfghanistanPeople> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Demography_of_Afghanistan> .
<http://dbpedia.org/resource/AfghanistanCommunications> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Communications_in_Afghanistan> .
<http://dbpedia.org/resource/AfghanistanMilitary> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Military_of_Afghanistan> .
<http://dbpedia.org/resource/AfghanistanTransportations> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Transport_in_Afghanistan> .
<http://dbpedia.org/resource/AfghanistanTransnationalIssues> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Foreign_relations_of_Afghanistan> .
<http://dbpedia.org/resource/AmoeboidTaxa> <http://dbpedia.org/ontology/wikiPageWikiLink> <http://dbpedia.org/resource/Amoeboid> .
Raw
Profiler output
Wed Jan 1 22:35 2014 Time and Allocation Profiling Report (Final)
reader +RTS -K1G -sstderr -pa -A3M -RTS
total time = 7.89 secs (7889 ticks @ 1000 us, 1 processor)
total alloc = 3,058,595,384 bytes (excludes profiling overheads)
COST CENTRE MODULE %time %alloc ticks bytes
GC GC 30.3 0.0 2387 0
popAndParse Main 13.6 11.5 1071 352000000
takeBytesWhile.go.(...) System.IO.Streams.ByteString 13.5 33.8 1066 1034405712
generalIndexer Main 10.9 9.8 856 301188920
resourceName Main 7.5 17.4 588 531200856
link Main 3.1 4.6 243 140799648
OVERHEAD_of PROFILING 2.2 0.0 177 0
readExactly.go.\ System.IO.Streams.ByteString 2.0 2.2 161 67199280
indexLinks.indexLoop Main 2.0 1.8 161 54399976
comment Main 2.0 4.0 156 121599960
takeBytesWhile.go System.IO.Streams.ByteString 1.5 1.2 117 35474240
indexLinks.insertLink Main 1.2 2.6 92 79999800
readExactly.go System.IO.Streams.ByteString 1.1 0.7 85 22400000
makeInputStream System.IO.Streams.Internal 1.0 1.3 79 38399928
takeBytesWhile.go.finish System.IO.Streams.ByteString 0.8 0.8 63 23181008
takeBytesWhile System.IO.Streams.ByteString 0.7 0.0 57 0
popAndParse.parsed Main 0.7 0.2 54 6400000
atEOF System.IO.Streams.Internal 0.7 0.0 52 0
atEOF.\ System.IO.Streams.Internal 0.7 0.3 52 9599976
handleToInputStream.f System.IO.Streams.Handle 0.7 1.9 52 57820344
linkLine Main 0.6 1.0 47 31999920
makeInputStream.pb System.IO.Streams.Internal 0.6 0.0 46 0
makeInputStream.grab System.IO.Streams.Internal 0.4 0.8 35 25599856
popLink Main 0.4 0.5 34 16000000
SYSTEM SYSTEM 0.4 0.0 31 16352
unRead System.IO.Streams.Internal 0.4 0.0 31 0
makeInputStream.pb.\ System.IO.Streams.Internal 0.3 1.3 27 38399784
linkLineParser Main 0.3 1.7 24 51200000
readExactly System.IO.Streams.ByteString 0.3 0.4 20 12800000
readExactly.go.\.l System.IO.Streams.ByteString 0.2 0.2 13 6400000
readExactly.go.\.(...) System.IO.Streams.ByteString 0.1 0.0 11 0
takeBytesWhile.go.dl' System.IO.Streams.ByteString 0.0 0.0 1 41136
IDLE IDLE 0.0 0.0 0 0
PINNED SYSTEM 0.0 0.0 0 0
DONT_CARE MAIN 0.0 0.0 0 0
MAIN MAIN 0.0 0.0 0 7000
CAF GHC.Integer.Type 0.0 0.0 0 0
CAF GHC.Integer.Logarithms.Internals 0.0 0.0 0 320
CAF GHC.IO.Encoding.Failure 0.0 0.0 0 0
CAF GHC.Event.PSQ 0.0 0.0 0 0
CAF GHC.IO.Encoding.UTF8 0.0 0.0 0 0
CAF GHC.IO.Encoding.UTF32 0.0 0.0 0 0
CAF GHC.IO.Encoding.UTF16 0.0 0.0 0 0
CAF GHC.Enum 0.0 0.0 0 0
CAF GHC.Event.Manager 0.0 0.0 0 0
CAF GHC.Event.Clock 0.0 0.0 0 0
CAF Foreign.Marshal.Alloc 0.0 0.0 0 0
CAF Data.Typeable.Internal 0.0 0.0 0 0
CAF GHC.Event.EPoll 0.0 0.0 0 0
CAF GHC.Int 0.0 0.0 0 0
CAF GHC.IO.Handle.Types 0.0 0.0 0 0
CAF GHC.IO.Encoding.Iconv 0.0 0.0 0 248
CAF GHC.Real 0.0 0.0 0 0
CAF GHC.Conc.Sync 0.0 0.0 0 0
CAF GHC.Event.Control 0.0 0.0 0 0
CAF System.Posix.Internals 0.0 0.0 0 0
CAF GHC.Show 0.0 0.0 0 0
CAF GHC.IO.Encoding 0.0 0.0 0 3376
CAF GHC.Float 0.0 0.0 0 0
CAF GHC.Exception 0.0 0.0 0 0
CAF GHC.Conc.Signal 0.0 0.0 0 672
CAF GHC.Arr 0.0 0.0 0 0
CAF GHC.Event.Thread 0.0 0.0 0 904
CAF GHC.Event.Internal 0.0 0.0 0 32
CAF Data.Maybe 0.0 0.0 0 0
CAF GHC.TopHandler 0.0 0.0 0 0
CAF GHC.List 0.0 0.0 0 0
CAF GHC.IO.Handle.Text 0.0 0.0 0 0
CAF GHC.IO.FD 0.0 0.0 0 32
CAF GHC.IO.Exception 0.0 0.0 0 0
CAF Control.Exception.Base 0.0 0.0 0 0
CAF GHC.IO.Handle.Internals 0.0 0.0 0 0
CAF GHC.IO.Handle.FD 0.0 0.0 0 34672
CAF GHC.IO.Handle 0.0 0.0 0 0
CAF GHC.ForeignPtr 0.0 0.0 0 0
CAF GHC.Err 0.0 0.0 0 0
CAF Data.ByteString.Internal 0.0 0.0 0 0
CAF Data.ByteString 0.0 0.0 0 0
CAF Data.Map 0.0 0.0 0 0
CAF Data.Attoparsec.ByteString.FastSet 0.0 0.0 0 0
CAF Data.Attoparsec.Internal.Types 0.0 0.0 0 0
CAF Data.Attoparsec.ByteString.Internal 0.0 0.0 0 0
CAF Control.Monad.IO.Class 0.0 0.0 0 0
CAF Data.Time.Clock.CTimeval 0.0 0.0 0 0
writeNonBlocking System.IO.Streams.Internal 0.0 0.0 0 0
write System.IO.Streams.Internal 0.0 0.0 0 0
readNonBlocking System.IO.Streams.Internal 0.0 0.0 0 0
read System.IO.Streams.Internal 0.0 0.0 0 0
read.f System.IO.Streams.Internal 0.0 0.0 0 0
read.f.\ System.IO.Streams.Internal 0.0 0.0 0 0
read.f.\.c System.IO.Streams.Internal 0.0 0.0 0 0
writeNonBlocking System.IO.Streams.Internal 0.0 0.0 0 0
write System.IO.Streams.Internal 0.0 0.0 0 0
readNonBlocking System.IO.Streams.Internal 0.0 0.0 0 0
read System.IO.Streams.Internal 0.0 0.0 0 0
writeNonBlocking System.IO.Streams.Internal 0.0 0.0 0 0
write System.IO.Streams.Internal 0.0 0.0 0 0
readNonBlocking System.IO.Streams.Internal 0.0 0.0 0 0
read System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer0 System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer0.l System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer0.s System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer System.IO.Streams.Internal 0.0 0.0 0 0
fillReadBuffer0 System.IO.Streams.Internal 0.0 0.0 0 0
fillReadBuffer System.IO.Streams.Internal 0.0 0.0 0 0
newBuffer System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer0 System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer System.IO.Streams.Internal 0.0 0.0 0 0
fillReadBuffer0 System.IO.Streams.Internal 0.0 0.0 0 0
fillReadBuffer System.IO.Streams.Internal 0.0 0.0 0 0
newBuffer System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer0 System.IO.Streams.Internal 0.0 0.0 0 0
flushWriteBuffer System.IO.Streams.Internal 0.0 0.0 0 0
fillReadBuffer0 System.IO.Streams.Internal 0.0 0.0 0 0
fillReadBuffer System.IO.Streams.Internal 0.0 0.0 0 0
newBuffer System.IO.Streams.Internal 0.0 0.0 0 0
devType System.IO.Streams.Internal 0.0 0.0 0 0
close System.IO.Streams.Internal 0.0 0.0 0 0
ready System.IO.Streams.Internal 0.0 0.0 0 0
devType System.IO.Streams.Internal 0.0 0.0 0 0
close System.IO.Streams.Internal 0.0 0.0 0 0
ready System.IO.Streams.Internal 0.0 0.0 0 0
devType System.IO.Streams.Internal 0.0 0.0 0 0
close System.IO.Streams.Internal 0.0 0.0 0 0
ready System.IO.Streams.Internal 0.0 0.0 0 0
return System.IO.Streams.Internal 0.0 0.0 0 0
>>= System.IO.Streams.Internal 0.0 0.0 0 0
liftIO System.IO.Streams.Internal 0.0 0.0 0 0
fmap System.IO.Streams.Internal 0.0 0.0 0 0
fmap.step System.IO.Streams.Internal 0.0 0.0 0 0
fmap.value System.IO.Streams.Internal 0.0 0.0 0 0
<*> System.IO.Streams.Internal 0.0 0.0 0 0
pure System.IO.Streams.Internal 0.0 0.0 0 0
return System.IO.Streams.Internal 0.0 0.0 0 0
>>= System.IO.Streams.Internal 0.0 0.0 0 0
>>=.step System.IO.Streams.Internal 0.0 0.0 0 0
>>=.value System.IO.Streams.Internal 0.0 0.0 0 0
liftIO System.IO.Streams.Internal 0.0 0.0 0 0
fmap System.IO.Streams.Internal 0.0 0.0 0 0
fmap.step System.IO.Streams.Internal 0.0 0.0 0 0
fmap.value System.IO.Streams.Internal 0.0 0.0 0 0
<*> System.IO.Streams.Internal 0.0 0.0 0 0
pure System.IO.Streams.Internal 0.0 0.0 0 0
typeOf2 System.IO.Streams.Internal 0.0 0.0 0 0
typeOf2 System.IO.Streams.Internal 0.0 0.0 0 0
typeOf1 System.IO.Streams.Internal 0.0 0.0 0 0
typeOf1 System.IO.Streams.Internal 0.0 0.0 0 0
typeOf2 System.IO.Streams.Internal 0.0 0.0 0 0
appendInputStream System.IO.Streams.Internal 0.0 0.0 0 0
concatInputStreams System.IO.Streams.Internal 0.0 0.0 0 0
concatInputStreams.run System.IO.Streams.Internal 0.0 0.0 0 0
concatInputStreams.run.go System.IO.Streams.Internal 0.0 0.0 0 0
lockingInputStream System.IO.Streams.Internal 0.0 0.0 0 0
lockingInputStream.grab System.IO.Streams.Internal 0.0 0.0 0 0
lockingInputStream.pb System.IO.Streams.Internal 0.0 0.0 0 0
supplyTo System.IO.Streams.Internal 0.0 0.0 0 0
supply System.IO.Streams.Internal 0.0 0.0 0 0
supply.loop System.IO.Streams.Internal 0.0 0.0 0 0
connectTo System.IO.Streams.Internal 0.0 0.0 0 0
connect System.IO.Streams.Internal 0.0 0.0 0 0
connect.loop System.IO.Streams.Internal 0.0 0.0 0 0
peek System.IO.Streams.Internal 0.0 0.0 0 0
read System.IO.Streams.Internal 0.0 0.0 0 0
lockingOutputStream System.IO.Streams.Internal 0.0 0.0 0 0
lockingOutputStream.f System.IO.Streams.Internal 0.0 0.0 0 0
write System.IO.Streams.Internal 0.0 0.0 0 0
fromGenerator System.IO.Streams.Internal 0.0 0.0 0 0
fromGenerator.go System.IO.Streams.Internal 0.0 0.0 0 0
fromGenerator.go.\ System.IO.Streams.Internal 0.0 0.0 0 0
fromGenerator.go.step System.IO.Streams.Internal 0.0 0.0 0 0
fromGenerator.go.finish System.IO.Streams.Internal 0.0 0.0 0 0
nullInput System.IO.Streams.Internal 0.0 0.0 0 0
fromConsumer System.IO.Streams.Internal 0.0 0.0 0 0
fromConsumer.go System.IO.Streams.Internal 0.0 0.0 0 0
fromConsumer.go.step System.IO.Streams.Internal 0.0 0.0 0 0
fromConsumer.go.force System.IO.Streams.Internal 0.0 0.0 0 0
nullOutput System.IO.Streams.Internal 0.0 0.0 0 0
makeOutputStream System.IO.Streams.Internal 0.0 0.0 0 0
bUFSIZ System.IO.Streams.Internal 0.0 0.0 0 0
unsupported System.IO.Streams.Internal 0.0 0.0 0 0
bufferToBS System.IO.Streams.Internal 0.0 0.0 0 0
bufferToBS.raw System.IO.Streams.Internal 0.0 0.0 0 0
bufferToBS.sz System.IO.Streams.Internal 0.0 0.0 0 0
bufferToBS.l System.IO.Streams.Internal 0.0 0.0 0 0
bufferToBS.r System.IO.Streams.Internal 0.0 0.0 0 0
emptyWriteBuffer System.IO.Streams.Internal 0.0 0.0 0 0
generatorBind System.IO.Streams.Internal 0.0 0.0 0 0
generatorBind.step System.IO.Streams.Internal 0.0 0.0 0 0
generatorBind.value System.IO.Streams.Internal 0.0 0.0 0 0
yield System.IO.Streams.Internal 0.0 0.0 0 0
await System.IO.Streams.Internal 0.0 0.0 0 0
CAF System.IO.Streams.Internal 0.0 0.0 0 32
fromList System.IO.Streams.List 0.0 0.0 0 0
fromList.f System.IO.Streams.List 0.0 0.0 0 0
fromList.f.\ System.IO.Streams.List 0.0 0.0 0 0
toList System.IO.Streams.List 0.0 0.0 0 0
outputToList System.IO.Streams.List 0.0 0.0 0 0
listOutputStream System.IO.Streams.List 0.0 0.0 0 0
listOutputStream.consumer System.IO.Streams.List 0.0 0.0 0 0
listOutputStream.consumer.go System.IO.Streams.List 0.0 0.0 0 0
listOutputStream.consumer.go.\ System.IO.Streams.List 0.0 0.0 0 0
listOutputStream.consumer.go.\.\ System.IO.Streams.List 0.0 0.0 0 0
listOutputStream.flush System.IO.Streams.List 0.0 0.0 0 0
listOutputStream.flush.\ System.IO.Streams.List 0.0 0.0 0 0
writeList System.IO.Streams.List 0.0 0.0 0 0
chunkList System.IO.Streams.List 0.0 0.0 0 0
chunkList.go System.IO.Streams.List 0.0 0.0 0 0
chunkList.go.finish System.IO.Streams.List 0.0 0.0 0 0
chunkList.go.finish.l System.IO.Streams.List 0.0 0.0 0 0
chunkList.go.chunk System.IO.Streams.List 0.0 0.0 0 0
concatLists System.IO.Streams.List 0.0 0.0 0 0
concatLists.chunk System.IO.Streams.List 0.0 0.0 0 0
concatLists.go System.IO.Streams.List 0.0 0.0 0 0
CAF System.IO.Streams.List 0.0 0.0 0 0
stdin System.IO.Streams.Handle 0.0 0.0 0 0
handleToInputStream System.IO.Streams.Handle 0.0 0.0 0 56
bUFSIZ System.IO.Streams.Handle 0.0 0.0 0 0
stderr System.IO.Streams.Handle 0.0 0.0 0 0
stdout System.IO.Streams.Handle 0.0 0.0 0 0
handleToOutputStream System.IO.Streams.Handle 0.0 0.0 0 0
handleToOutputStream.f System.IO.Streams.Handle 0.0 0.0 0 0
inputStreamToHandle System.IO.Streams.Handle 0.0 0.0 0 0
outputStreamToHandle System.IO.Streams.Handle 0.0 0.0 0 0
streamPairToHandle System.IO.Streams.Handle 0.0 0.0 0 0
CAF System.IO.Streams.Handle 0.0 0.0 0 32
outputFoldM System.IO.Streams.Combinators 0.0 0.0 0 0
outputFoldM.wr System.IO.Streams.Combinators 0.0 0.0 0 0
outputFoldM.fetch System.IO.Streams.Combinators 0.0 0.0 0 0
outputFoldM.fetch.\ System.IO.Streams.Combinators 0.0 0.0 0 0
inputFoldM System.IO.Streams.Combinators 0.0 0.0 0 0
inputFoldM.rd System.IO.Streams.Combinators 0.0 0.0 0 0
inputFoldM.twiddle System.IO.Streams.Combinators 0.0 0.0 0 0
inputFoldM.fetch System.IO.Streams.Combinators 0.0 0.0 0 0
inputFoldM.fetch.\ System.IO.Streams.Combinators 0.0 0.0 0 0
fold System.IO.Streams.Combinators 0.0 0.0 0 0
fold.go System.IO.Streams.Combinators 0.0 0.0 0 0
foldM System.IO.Streams.Combinators 0.0 0.0 0 0
foldM.go System.IO.Streams.Combinators 0.0 0.0 0 0
any System.IO.Streams.Combinators 0.0 0.0 0 0
any.go System.IO.Streams.Combinators 0.0 0.0 0 0
all System.IO.Streams.Combinators 0.0 0.0 0 0
all.go System.IO.Streams.Combinators 0.0 0.0 0 0
maximum System.IO.Streams.Combinators 0.0 0.0 0 0
maximum.go System.IO.Streams.Combinators 0.0 0.0 0 0
minimum System.IO.Streams.Combinators 0.0 0.0 0 0
minimum.go System.IO.Streams.Combinators 0.0 0.0 0 0
unfoldM System.IO.Streams.Combinators 0.0 0.0 0 0
unfoldM.go System.IO.Streams.Combinators 0.0 0.0 0 0
map System.IO.Streams.Combinators 0.0 0.0 0 0
map.g System.IO.Streams.Combinators 0.0 0.0 0 0
mapM System.IO.Streams.Combinators 0.0 0.0 0 0
mapM.g System.IO.Streams.Combinators 0.0 0.0 0 0
mapM.g.\ System.IO.Streams.Combinators 0.0 0.0 0 0
mapM_ System.IO.Streams.Combinators 0.0 0.0 0 0
contramap System.IO.Streams.Combinators 0.0 0.0 0 0
contramapM System.IO.Streams.Combinators 0.0 0.0 0 0
contramapM.g System.IO.Streams.Combinators 0.0 0.0 0 0
contramapM_ System.IO.Streams.Combinators 0.0 0.0 0 0
contramapM_.\ System.IO.Streams.Combinators 0.0 0.0 0 0
skipToEof System.IO.Streams.Combinators 0.0 0.0 0 0
skipToEof.go System.IO.Streams.Combinators 0.0 0.0 0 0
filterM System.IO.Streams.Combinators 0.0 0.0 0 0
filterM.chunk System.IO.Streams.Combinators 0.0 0.0 0 0
filterM.prod System.IO.Streams.Combinators 0.0 0.0 0 0
filterM.eof System.IO.Streams.Combinators 0.0 0.0 0 0
filterM.pb System.IO.Streams.Combinators 0.0 0.0 0 0
filter System.IO.Streams.Combinators 0.0 0.0 0 0
filter.chunk System.IO.Streams.Combinators 0.0 0.0 0 0
filter.chunk.b System.IO.Streams.Combinators 0.0 0.0 0 0
filter.prod System.IO.Streams.Combinators 0.0 0.0 0 0
filter.eof System.IO.Streams.Combinators 0.0 0.0 0 0
filter.pb System.IO.Streams.Combinators 0.0 0.0 0 0
intersperse System.IO.Streams.Combinators 0.0 0.0 0 0
intersperse.f System.IO.Streams.Combinators 0.0 0.0 0 0
zip System.IO.Streams.Combinators 0.0 0.0 0 0
zip.src System.IO.Streams.Combinators 0.0 0.0 0 0
zip.src.\ System.IO.Streams.Combinators 0.0 0.0 0 0
zip.src.\.\ System.IO.Streams.Combinators 0.0 0.0 0 0
zipWith System.IO.Streams.Combinators 0.0 0.0 0 0
zipWith.src System.IO.Streams.Combinators 0.0 0.0 0 0
zipWith.src.\ System.IO.Streams.Combinators 0.0 0.0 0 0
zipWith.src.\.\ System.IO.Streams.Combinators 0.0 0.0 0 0
zipWithM System.IO.Streams.Combinators 0.0 0.0 0 0
zipWithM.src System.IO.Streams.Combinators 0.0 0.0 0 0
zipWithM.src.\ System.IO.Streams.Combinators 0.0 0.0 0 0
zipWithM.src.\.\ System.IO.Streams.Combinators 0.0 0.0 0 0
zipWithM.src.\.\.\ System.IO.Streams.Combinators 0.0 0.0 0 0
filterOutput System.IO.Streams.Combinators 0.0 0.0 0 0
filterOutput.chunk System.IO.Streams.Combinators 0.0 0.0 0 0
filterOutputM System.IO.Streams.Combinators 0.0 0.0 0 0
filterOutputM.chunk System.IO.Streams.Combinators 0.0 0.0 0 0
unzip System.IO.Streams.Combinators 0.0 0.0 0 0
unzip.twist System.IO.Streams.Combinators 0.0 0.0 0 0
unzip.src System.IO.Streams.Combinators 0.0 0.0 0 0
unzip.src.more System.IO.Streams.Combinators 0.0 0.0 0 0
unzip.src.more.\ System.IO.Streams.Combinators 0.0 0.0 0 0
unzip.src.more.\.b System.IO.Streams.Combinators 0.0 0.0 0 0
unzip.src.more.\.a System.IO.Streams.Combinators 0.0 0.0 0 0
unzip.src.more.\.(...) System.IO.Streams.Combinators 0.0 0.0 0 0
take System.IO.Streams.Combinators 0.0 0.0 0 0
take.prod System.IO.Streams.Combinators 0.0 0.0 0 0
take.prod.\ System.IO.Streams.Combinators 0.0 0.0 0 0
take.pb System.IO.Streams.Combinators 0.0 0.0 0 0
drop System.IO.Streams.Combinators 0.0 0.0 0 0
drop.discard System.IO.Streams.Combinators 0.0 0.0 0 0
drop.prod System.IO.Streams.Combinators 0.0 0.0 0 0
drop.getInput System.IO.Streams.Combinators 0.0 0.0 0 0
drop.getInput.\ System.IO.Streams.Combinators 0.0 0.0 0 0
drop.getInput.\.\ System.IO.Streams.Combinators 0.0 0.0 0 0
drop.pb System.IO.Streams.Combinators 0.0 0.0 0 0
give System.IO.Streams.Combinators 0.0 0.0 0 0
give.chunk System.IO.Streams.Combinators 0.0 0.0 0 0
give.chunk.\ System.IO.Streams.Combinators 0.0 0.0 0 0
ignore System.IO.Streams.Combinators 0.0 0.0 0 0
ignore.chunk System.IO.Streams.Combinators 0.0 0.0 0 0
ignore.chunk.\ System.IO.Streams.Combinators 0.0 0.0 0 0
ignoreEof System.IO.Streams.Combinators 0.0 0.0 0 0
ignoreEof.f System.IO.Streams.Combinators 0.0 0.0 0 0
atEndOfInput System.IO.Streams.Combinators 0.0 0.0 0 0
atEndOfInput.prod System.IO.Streams.Combinators 0.0 0.0 0 0
atEndOfInput.eof System.IO.Streams.Combinators 0.0 0.0 0 0
atEndOfInput.pb System.IO.Streams.Combinators 0.0 0.0 0 0
atEndOfOutput System.IO.Streams.Combinators 0.0 0.0 0 0
atEndOfOutput.f System.IO.Streams.Combinators 0.0 0.0 0 0
CAF System.IO.Streams.Combinators 0.0 0.0 0 0
show System.IO.Streams.ByteString 0.0 0.0 0 0
show System.IO.Streams.ByteString 0.0 0.0 0 0
show System.IO.Streams.ByteString 0.0 0.0 0 0
show System.IO.Streams.ByteString 0.0 0.0 0 0
show System.IO.Streams.ByteString 0.0 0.0 0 0
typeOf System.IO.Streams.ByteString 0.0 0.0 0 0
typeOf System.IO.Streams.ByteString 0.0 0.0 0 0
typeOf System.IO.Streams.ByteString 0.0 0.0 0 0
typeOf System.IO.Streams.ByteString 0.0 0.0 0 0
typeOf System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow.prod System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow.prod.chunk System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow.prod.chunk.newBytes System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow.prod.chunk.slen System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow.pb System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow.pb.\ System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfTooSlow.minSeconds System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.prod System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.prod.chunk System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.prod.chunk.b System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.prod.chunk.a System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.prod.chunk.(...) System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.prod.chunk.k' System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.prod.chunk.l System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfProducesMoreThan.pb System.IO.Streams.ByteString 0.0 0.0 0 0
words System.IO.Streams.ByteString 0.0 0.0 0 0
lines System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start.chunk System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start.chunk.\ System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start.chunk.b System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start.chunk.a System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start.chunk.(...) System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start.go System.IO.Streams.ByteString 0.0 0.0 0 0
splitOn.start.end System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.prod System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.prod.chunk System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.prod.chunk.b System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.prod.chunk.a System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.prod.chunk.(...) System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.prod.chunk.k' System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.prod.chunk.l System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytes.pb System.IO.Streams.ByteString 0.0 0.0 0 0
countInput System.IO.Streams.ByteString 0.0 0.0 0 0
countInput.prod System.IO.Streams.ByteString 0.0 0.0 0 0
countInput.prod.\ System.IO.Streams.ByteString 0.0 0.0 0 0
countInput.pb System.IO.Streams.ByteString 0.0 0.0 0 0
countInput.pb.\ System.IO.Streams.ByteString 0.0 0.0 0 0
modifyRef System.IO.Streams.ByteString 0.0 0.0 0 0
writeLazyByteString System.IO.Streams.ByteString 0.0 0.0 0 0
fromByteString System.IO.Streams.ByteString 0.0 0.0 0 0
fromLazyByteString System.IO.Streams.ByteString 0.0 0.0 0 0
countOutput System.IO.Streams.ByteString 0.0 0.0 0 0
countOutput.f System.IO.Streams.ByteString 0.0 0.0 0 0
unlines System.IO.Streams.ByteString 0.0 0.0 0 0
unlines.\ System.IO.Streams.ByteString 0.0 0.0 0 0
unwords System.IO.Streams.ByteString 0.0 0.0 0 0
readExactly.go.\.b System.IO.Streams.ByteString 0.0 0.0 0 0
readExactly.go.\.a System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytesWhile.go.b System.IO.Streams.ByteString 0.0 0.0 0 0
takeBytesWhile.go.a System.IO.Streams.ByteString 0.0 0.0 0 0
giveBytes System.IO.Streams.ByteString 0.0 0.0 0 0
giveBytes.sink System.IO.Streams.ByteString 0.0 0.0 0 0
giveBytes.sink.a System.IO.Streams.ByteString 0.0 0.0 0 0
giveBytes.sink.k' System.IO.Streams.ByteString 0.0 0.0 0 0
giveBytes.sink.l System.IO.Streams.ByteString 0.0 0.0 0 0
giveExactly System.IO.Streams.ByteString 0.0 0.0 0 0
giveExactly.go System.IO.Streams.ByteString 0.0 0.0 0 0
giveExactly.go.n' System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfConsumesMoreThan System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfConsumesMoreThan.sink System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfConsumesMoreThan.sink.k' System.IO.Streams.ByteString 0.0 0.0 0 0
throwIfConsumesMoreThan.sink.l System.IO.Streams.ByteString 0.0 0.0 0 0
getTime System.IO.Streams.ByteString 0.0 0.0 0 0
CAF System.IO.Streams.ByteString 0.0 0.0 0 0
rnf Main 0.0 0.0 0 0
rnf Main 0.0 0.0 0 0
showList Main 0.0 0.0 0 0
showsPrec Main 0.0 0.0 0 0
showList Main 0.0 0.0 0 0
showsPrec Main 0.0 0.0 0 0
showList Main 0.0 0.0 0 0
showsPrec Main 0.0 0.0 0 0
main Main 0.0 0.0 0 19520
indexLinks Main 0.0 0.0 0 200
linkspath Main 0.0 0.0 0 1256
CAF Main 0.0 0.0 0 336
individual inherited
COST CENTRE MODULE no. entries %time %alloc %time %alloc ticks bytes
MAIN MAIN 61 0 0.0 0.0 100.0 100.0 0 7000
main Main 123 0 0.0 0.0 67.1 100.0 0 19520
indexLinks Main 125 1 0.0 0.0 67.1 100.0 0 200
atEOF.\ System.IO.Streams.Internal 139 1 0.0 0.0 0.0 0.0 0 0
read System.IO.Streams.Internal 130 1 0.0 0.0 0.0 0.0 0 0
atEOF System.IO.Streams.Internal 129 1 0.0 0.0 0.0 0.0 0 0
indexLinks.indexLoop Main 128 400001 2.0 1.8 67.1 100.0 161 54399976
atEOF.\ System.IO.Streams.Internal 199 399999 0.0 0.0 0.0 0.0 0 0
read System.IO.Streams.Internal 198 400000 0.0 0.0 0.0 0.0 0 0
indexLinks.insertLink Main 197 400000 1.2 2.6 12.0 12.5 92 79999800
generalIndexer Main 206 799998 10.9 9.8 10.9 9.8 856 301188920
popLink Main 148 0 0.4 0.5 51.8 84.6 34 16000000
popAndParse Main 149 400000 13.6 11.5 51.3 84.1 1071 352000000
popAndParse.parsed Main 192 400000 0.7 0.2 14.1 28.9 54 6400000
linkLineParser Main 194 0 0.3 1.7 13.4 28.7 24 51200000
comment Main 196 0 2.0 4.0 13.1 27.0 156 121599960
linkLine Main 201 0 0.6 1.0 11.1 23.0 47 31999920
resourceName Main 203 0 7.5 17.4 10.5 22.0 588 531198672
link Main 205 0 3.1 4.6 3.1 4.6 243 140799648
readExactly System.IO.Streams.ByteString 176 400000 0.3 0.4 4.3 4.4 20 12800000
readExactly.go System.IO.Streams.ByteString 177 400000 1.1 0.7 4.0 4.0 85 22400000
readExactly.go.\ System.IO.Streams.ByteString 182 400000 2.0 2.2 2.8 3.0 161 67199280
readExactly.go.\.a System.IO.Streams.ByteString 191 400000 0.0 0.0 0.0 0.0 0 0
unRead System.IO.Streams.Internal 186 0 0.0 0.0 0.5 0.6 3 0
handleToInputStream System.IO.Streams.Handle 187 0 0.0 0.0 0.4 0.6 0 0
makeInputStream System.IO.Streams.Internal 188 0 0.3 0.3 0.4 0.6 21 9599784
makeInputStream.pb System.IO.Streams.Internal 189 399991 0.1 0.0 0.2 0.3 7 0
makeInputStream.pb.\ System.IO.Streams.Internal 190 399991 0.1 0.3 0.1 0.3 5 9599784
readExactly.go.\.(...) System.IO.Streams.ByteString 185 400000 0.1 0.0 0.1 0.0 11 0
readExactly.go.\.b System.IO.Streams.ByteString 184 400000 0.0 0.0 0.0 0.0 0 0
readExactly.go.\.l System.IO.Streams.ByteString 183 400000 0.2 0.2 0.2 0.2 13 6400000
handleToInputStream System.IO.Streams.Handle 179 0 0.0 0.0 0.2 0.2 0 0
makeInputStream System.IO.Streams.Internal 180 0 0.1 0.0 0.2 0.2 4 0
makeInputStream.grab System.IO.Streams.Internal 181 0 0.1 0.2 0.1 0.2 8 6400000
read System.IO.Streams.Internal 178 400000 0.0 0.0 0.0 0.0 0 0
atEOF System.IO.Streams.Internal 165 400000 0.5 0.0 1.2 0.8 37 0
atEOF.\ System.IO.Streams.Internal 170 400000 0.2 0.0 0.6 0.6 15 0
unRead System.IO.Streams.Internal 171 0 0.0 0.0 0.4 0.6 2 0
handleToInputStream System.IO.Streams.Handle 172 0 0.0 0.0 0.4 0.6 0 0
makeInputStream System.IO.Streams.Internal 173 0 0.1 0.3 0.4 0.6 10 9600000
makeInputStream.pb System.IO.Streams.Internal 174 400000 0.2 0.0 0.3 0.3 15 0
makeInputStream.pb.\ System.IO.Streams.Internal 175 400000 0.1 0.3 0.1 0.3 8 9600000
handleToInputStream System.IO.Streams.Handle 167 0 0.0 0.0 0.1 0.2 0 0
makeInputStream System.IO.Streams.Internal 168 0 0.1 0.0 0.1 0.2 7 0
makeInputStream.grab System.IO.Streams.Internal 169 0 0.1 0.2 0.1 0.2 4 6400000
read System.IO.Streams.Internal 166 400000 0.0 0.0 0.0 0.0 0 0
takeBytesWhile System.IO.Streams.ByteString 150 400000 0.7 0.0 18.1 38.5 57 0
takeBytesWhile.go System.IO.Streams.ByteString 155 401714 1.5 1.2 17.1 38.2 117 35474240
takeBytesWhile.go.dl' System.IO.Streams.ByteString 212 1714 0.0 0.0 0.0 0.0 0 0
handleToInputStream System.IO.Streams.Handle 208 0 0.0 0.0 0.7 1.9 0 0
handleToInputStream.f System.IO.Streams.Handle 211 0 0.7 1.9 0.7 1.9 52 57485032
makeInputStream System.IO.Streams.Internal 209 0 0.0 0.0 0.0 0.0 0 0
makeInputStream.grab System.IO.Streams.Internal 210 0 0.0 0.0 0.0 0.0 0 0
read System.IO.Streams.Internal 207 1714 0.0 0.0 0.0 0.0 0 0
takeBytesWhile.go.a System.IO.Streams.ByteString 164 401714 0.0 0.0 0.0 0.0 0 0
takeBytesWhile.go.finish System.IO.Streams.ByteString 163 400000 0.8 0.8 0.8 0.8 63 23181008
takeBytesWhile.go.dl' System.IO.Streams.ByteString 213 0 0.0 0.0 0.0 0.0 1 41136
unRead System.IO.Streams.Internal 158 0 0.2 0.0 0.7 0.6 19 0
handleToInputStream System.IO.Streams.Handle 159 0 0.0 0.0 0.4 0.6 0 0
makeInputStream System.IO.Streams.Internal 160 0 0.1 0.3 0.4 0.6 11 9600000
makeInputStream.pb System.IO.Streams.Internal 161 400000 0.2 0.0 0.3 0.3 12 0
makeInputStream.pb.\ System.IO.Streams.Internal 162 400000 0.1 0.3 0.1 0.3 11 9600000
takeBytesWhile.go.b System.IO.Streams.ByteString 157 401714 0.0 0.0 0.0 0.0 0 0
takeBytesWhile.go.(...) System.IO.Streams.ByteString 156 401714 13.5 33.8 13.5 33.8 1066 1034405712
handleToInputStream System.IO.Streams.Handle 152 0 0.0 0.0 0.3 0.2 0 0
makeInputStream System.IO.Streams.Internal 153 0 0.1 0.0 0.3 0.2 4 0
makeInputStream.grab System.IO.Streams.Internal 154 0 0.2 0.2 0.2 0.2 18 6400000
read System.IO.Streams.Internal 151 400000 0.0 0.0 0.0 0.0 0 0
atEOF System.IO.Streams.Internal 131 400000 0.2 0.0 1.3 1.2 15 0
atEOF.\ System.IO.Streams.Internal 140 0 0.5 0.3 1.0 0.9 37 9599976
unRead System.IO.Streams.Internal 142 0 0.1 0.0 0.5 0.6 7 0
handleToInputStream System.IO.Streams.Handle 143 0 0.0 0.0 0.4 0.6 0 0
makeInputStream System.IO.Streams.Internal 144 0 0.2 0.3 0.4 0.6 16 9600000
makeInputStream.pb System.IO.Streams.Internal 145 400000 0.2 0.0 0.2 0.3 12 0
makeInputStream.pb.\ System.IO.Streams.Internal 146 400000 0.0 0.3 0.0 0.3 3 9600000
handleToInputStream System.IO.Streams.Handle 133 0 0.0 0.0 0.1 0.2 0 0
handleToInputStream.f System.IO.Streams.Handle 137 0 0.0 0.0 0.0 0.0 0 335312
makeInputStream System.IO.Streams.Internal 134 0 0.1 0.0 0.1 0.2 6 0
makeInputStream.grab System.IO.Streams.Internal 135 0 0.1 0.2 0.1 0.2 5 6399856
handleToInputStream System.IO.Streams.Handle 126 1 0.0 0.0 0.0 0.0 0 56
handleToInputStream.f System.IO.Streams.Handle 136 1 0.0 0.0 0.0 0.0 0 0
makeInputStream System.IO.Streams.Internal 127 1 0.0 0.0 0.0 0.0 0 144
makeInputStream.grab System.IO.Streams.Internal 132 1 0.0 0.0 0.0 0.0 0 0
CAF Main 121 0 0.0 0.0 0.0 0.0 0 336
link Main 204 1 0.0 0.0 0.0 0.0 0 0
resourceName Main 202 1 0.0 0.0 0.0 0.0 0 2184
linkLine Main 200 1 0.0 0.0 0.0 0.0 0 0
comment Main 195 1 0.0 0.0 0.0 0.0 0 0
linkLineParser Main 193 1 0.0 0.0 0.0 0.0 0 0
popLink Main 147 1 0.0 0.0 0.0 0.0 0 0
linkspath Main 124 1 0.0 0.0 0.0 0.0 0 1256
main Main 122 1 0.0 0.0 0.0 0.0 0 0
CAF System.IO.Streams.ByteString 120 0 0.0 0.0 0.0 0.0 0 0
CAF System.IO.Streams.Combinators 119 0 0.0 0.0 0.0 0.0 0 0
CAF System.IO.Streams.Handle 118 0 0.0 0.0 0.0 0.0 0 32
bUFSIZ System.IO.Streams.Handle 138 1 0.0 0.0 0.0 0.0 0 0
CAF System.IO.Streams.List 117 0 0.0 0.0 0.0 0.0 0 0
CAF System.IO.Streams.Internal 116 0 0.0 0.0 0.0 0.0 0 32
unRead System.IO.Streams.Internal 141 1 0.0 0.0 0.0 0.0 0 0
CAF Data.Time.Clock.CTimeval 115 0 0.0 0.0 0.0 0.0 0 0
CAF Control.Monad.IO.Class 114 0 0.0 0.0 0.0 0.0 0 0
CAF Data.Attoparsec.ByteString.Internal 113 0 0.0 0.0 0.0 0.0 0 0
CAF Data.Attoparsec.Internal.Types 112 0 0.0 0.0 0.0 0.0 0 0
CAF Data.Attoparsec.ByteString.FastSet 111 0 0.0 0.0 0.0 0.0 0 0
CAF Data.Map 110 0 0.0 0.0 0.0 0.0 0 0
CAF Data.ByteString 109 0 0.0 0.0 0.0 0.0 0 0
CAF Data.ByteString.Internal 108 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Err 107 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.ForeignPtr 106 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Handle 105 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Handle.FD 104 0 0.0 0.0 0.0 0.0 0 34672
CAF GHC.IO.Handle.Internals 103 0 0.0 0.0 0.0 0.0 0 0
CAF Control.Exception.Base 102 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Exception 101 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.FD 100 0 0.0 0.0 0.0 0.0 0 32
CAF GHC.IO.Handle.Text 99 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.List 98 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.TopHandler 97 0 0.0 0.0 0.0 0.0 0 0
CAF Data.Maybe 96 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Event.Internal 95 0 0.0 0.0 0.0 0.0 0 32
CAF GHC.Event.Thread 94 0 0.0 0.0 0.0 0.0 0 904
CAF GHC.Arr 93 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Conc.Signal 92 0 0.0 0.0 0.0 0.0 0 672
CAF GHC.Exception 91 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Float 90 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Encoding 89 0 0.0 0.0 0.0 0.0 0 3376
CAF GHC.Show 88 0 0.0 0.0 0.0 0.0 0 0
CAF System.Posix.Internals 87 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Event.Control 86 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Conc.Sync 85 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Real 84 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Encoding.Iconv 83 0 0.0 0.0 0.0 0.0 0 248
CAF GHC.IO.Handle.Types 82 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Int 81 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Event.EPoll 80 0 0.0 0.0 0.0 0.0 0 0
CAF Data.Typeable.Internal 79 0 0.0 0.0 0.0 0.0 0 0
CAF Foreign.Marshal.Alloc 78 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Event.Clock 77 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Event.Manager 76 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Enum 75 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Encoding.UTF16 74 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Encoding.UTF32 73 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Encoding.UTF8 72 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Event.PSQ 71 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.IO.Encoding.Failure 70 0 0.0 0.0 0.0 0.0 0 0
CAF GHC.Integer.Logarithms.Internals 69 0 0.0 0.0 0.0 0.0 0 320
CAF GHC.Integer.Type 68 0 0.0 0.0 0.0 0.0 0 0
SYSTEM SYSTEM 67 0 0.4 0.0 0.4 0.0 31 16352
GC GC 66 0 30.3 0.0 30.3 0.0 2387 0
OVERHEAD_of PROFILING 65 0 2.2 0.0 2.2 0.0 177 0
DONT_CARE MAIN 64 0 0.0 0.0 0.0 0.0 0 0
PINNED SYSTEM 63 0 0.0 0.0 0.0 0.0 0 0
IDLE IDLE 62 0 0.0 0.0 0.0 0.0 0 0
Raw
reader.hs
{-# LANGUAGE OverloadedStrings #-}
import Prelude hiding (takeWhile, take)
import Data.Attoparsec.Char8
import Control.Applicative
import qualified Data.ByteString.Char8 as BS
import System.IO hiding (hGetLine)
import Control.Monad.Loops
import Control.Monad
import qualified Data.Map as M
import Data.Maybe
import Data.Hashable
import Control.Parallel
import System.Random
import Debug.Trace
import qualified Data.Foldable (sum)
import Data.List (foldl')
import Data.IORef
import Data.HashTable as HT
import Control.DeepSeq
import Data.Int
import System.IO.Streams.ByteString
import System.IO.Streams.Handle
import System.IO.Streams (read, atEOF, InputStream)
data NTEntry = NTEntry Resource Schema Content deriving Show
data NTLink = NTLink From To deriving Show
type From = BS.ByteString
type To = BS.ByteString
type Resource = BS.ByteString
type Schema = BS.ByteString
type Content = BS.ByteString
instance NFData NTEntry where
rnf (NTEntry r s c) = (rnf r) `seq` (rnf s) `seq` (rnf c) `seq` ()
instance NFData BS.ByteString where
rnf bs = bs `seq` ()
type Handlemap = M.Map Int [Integer]
data TypedHandleMap a = TypedHandleMap (M.Map Int [Integer]) deriving Show
--instance Hashable BS.ByteString where
--hash = hash . BS.unpack
type IXTable = HT.HashTable Resource [Integer]
main = do
f <- openFile linkspath ReadMode
ixtable <- indexLinks f
ixls <- HT.toList ixtable
putStrLn $ show $ ixls
indexLinks :: Handle -> IO IXTable
indexLinks f = do
table <- (HT.new (==) (fromIntegral . hash)) :: IO (IXTable)
fStream <- handleToInputStream f
indexLoop f fStream table
return table
where
indexLoop ::
Handle ->
InputStream BS.ByteString ->
IXTable ->
IO ()
indexLoop h stream table = do
ended <- atEOF stream
if ended
then return ()
else do
v <- popLink h stream
insertLink table v
indexLoop h stream table
insertLink ::
IXTable
-> (Maybe (Integer, Maybe NTLink))
-> IO ()
insertLink _ Nothing = return ()
insertLink _ (Just (_, Nothing)) = return ()
insertLink table (Just (pos, Just (NTLink f t))) = do
generalIndexer table f pos
generalIndexer table t pos
generalIndexer ::
IXTable
-> Resource -> Integer
-> IO ()
generalIndexer index b pos = do
l <- HT.lookup index b
case l of
Nothing -> insert index b [pos]
Just ls -> insert index b (pos : ls)
linkspath = "page_links_en.nt_"
abstractspath = "short_abstracts_en.nt"
popAndParse :: Parser a ->
Handle ->
InputStream BS.ByteString ->
IO (Maybe (Integer, a))
popAndParse parser h stream = do
lineStart <- hTell h
line <- takeBytesWhile ((/=) '\n') stream
case line of
Nothing -> return Nothing
Just l -> do
ended <- atEOF stream
if ended
then return ()
else do
readExactly 1 stream
return ()
let parsed = parseOnly parser l in
parsed `par` case parsed of
Left error -> do
hPutStrLn System.IO.stderr $ show error
return $ Nothing
Right v ->
return $ Just (lineStart, v)
popAbstract = popAndParse abstractParser
popLink = popAndParse linkLineParser
linkLineParser :: Parser (Maybe NTLink)
linkLineParser =
(comment >> return Nothing)
<|> linkLine
linkLine = do
char '<'
from <- resourceName
char '>'
skipWhile isSpace
char '<'
link
char '>'
skipWhile isSpace
char '<'
to <- resourceName
char '>'
skipWhile ((/=) '\n')
return . Just $ NTLink from to
resourceName = do
string "http://dbpedia.org/resource/"
takeWhile ((/= '>'))
abstractParser :: Parser (Maybe NTEntry)
abstractParser =
(comment >> return Nothing)
<|> entry
comment :: Parser ()
comment = do
skipWhile isSpace
char '#'
skipWhile ((/=) '\n')
entry :: Parser (Maybe NTEntry)
entry = do
skipWhile isSpace
char '<'
resource <- resourceName
char '>'
skipWhile isSpace
char '<'
schema <- link
char '>'
skipWhile isSpace
char '"'
content <- quotedString
char '"'
char '@'
lang <- language
skipWhile ((/=) '\n')
case lang of
"en" -> return $ Just $ NTEntry resource schema content
_ -> return Nothing
link = takeWhile ((/=) '>')
quotedString = takeWhile ((/=) '"')
language = take 2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment