Steps to generate Dgraphs fulltext index tokens

Install Go
Download Dgraph repo using go get
```
go get -u github.com/dgraph-io/dgraph
```
Go the tok package directory inside Dgraph's source.
```
cd $GOPATH/src/github.com/dgraph-io/dgraph/tok
```
Turn the Go modules on
```
export GO111MODULE=on
```

Copy the following test into the tok_test.go file.

package tok

import (
        "testing"
        "github.com/stretchr/testify/require"
)


func TestGetTokensForFullText(t *testing.T) {
        val := "Let's Go and catch @francesc at @Gopherpalooza today, as he scans into Go source code by building its Graph in Dgraph!\nBe there, as he Goes through analyzing Go source code, using a Go program, that stores data in the GraphDB built in Go!\n#golang #GraphDB #Databases #Dgraph"
         // Call the fulltext tokenizer.
        tokens, err := (&FullTextTokenizer{lang: "en"}).Tokens(val)
        require.NoError(t, err)
        // Print the generated tokens.
        t.Logf("%v",tokens)
}

Now, run the test, this should print the fulltext tokens for the matched tweet. Change the value of the val variable in the above test to find the fulltext tokens for a diffrent sentence or a text paragraph.
```
go test -run=TestGetTokensForFullText -v
```

hackintoshrao/generate_fulltext_token.md