Skip to content

Instantly share code, notes, and snippets.

@gregtap
Created August 5, 2016 14:44
Show Gist options
  • Select an option

  • Save gregtap/0c19c63bad1124425dab0dece4dbf908 to your computer and use it in GitHub Desktop.

Select an option

Save gregtap/0c19c63bad1124425dab0dece4dbf908 to your computer and use it in GitHub Desktop.
Test go
type WorkUnit struct {
Urls [1000] string
}
accUrls := make([]string, 1000)
var url string
i := 1
for rows.Next() {
err := rows.Scan(&url)
panicOnErr(err)
if i%100000 == 0 {
fmt.Printf("%vK\n", i/1000)
}
i += 1
_url := "/" + url
accUrls = accUrls.append(_url)
// Every 1K urls build a unit of work.
if i%1000 == 0 {
work := WorkUnit{Urls: accUrls} // ERRROR type []string) as type [1000]string in assignment
// clear urls.
accUrls = accUrls[:0]
// Pass unit of work to work queue.
WorkQueue <- work
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment