gotokenizer

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

APACHE-2.0 License

Stars

Ecosystems: Go

Total Pull Requests

Merged Pull Requests

Total Issues

Time to Close Issues

N/A

Package Rankings

Top 7.57% on Proxy.golang.org

Badges

Extracted from project README

WebAssembly efficient text segmentation; support english, chinese, japanese and other. WebAssembl...

"结巴"中文分词的Golang版本

grep for words with similar meaning to the query

Tokenizers and lemmatizers for Go

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and grap...

Go efficient multilingual NLP and text segmentation; support English, Chinese, Japanese and others.

A multilingual command line sentence tokenizer in Golang

基于 Trie 的敏感词检查 🤓

package lingo provides the data structures and algorithms required for natural language processing

OpenAI ChatGPT, GPT-3, GPT-4, DALL·E, Whisper API wrapper for Go