Calculate the frequencies of words, pairs of words and more in a Wikipedia datas...
source link: https://github.com/asimihsan/word-frequencies
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
word-frequencies
Calculate the frequencies of words, pairs of words, etc. in a Wikipedia dataset. One use-case is to create lists of popular, different words to use in e.g. games, passphrase generation, etc.
Installation
This crate is not published to crates.io yet so you will need to first install Rust , clone this repository locally, then run:
cargo install --path . --force
This will put a word-frequencies
binary into your $HOME/.cargo/bin
folder, which you can then put into your PATH
environment variable.
Usage
Run word-frequencies --help
and e.g. word-frequencies split --help
for usage instructions. Below is an end-to-end example of
using word-frequencies
to count unigrams (words) and bigrams (pairs of words), and then calculate the most frequent words.
1. Wikipedia dataset download
First download the Wikipedia dataset for the language that you care about.
- Browse to the Wikipedia CirrusSearch dataset dumps: https://dumps.wikimedia.org/other/cirrussearch/
-
Browse to last complete download. To be safe choose the date before the current one.
- In this example it was https://dumps.wikimedia.org/other/cirrussearch/20200113/
-
Download the "content" file for the language you care about.
- In this example English is https://dumps.wikimedia.org/other/cirrussearch/20200113/enwiki-20200113-cirrussearch-content.json.gz
- In this example Polish is https://dumps.wikimedia.org/other/cirrussearch/20200113/plwiki-20200113-cirrussearch-content.json.gz
2a. Mac and Linux command-line example
Let's assume the Wikipedia dataset is downloaded to $HOME/datasets/wikipedia/plwiki-20200113-cirrussearch-content.json.gz
.
With this file downloaded, first split the file into multiple pieces, and also Unicode-normalize the input. The output files will be one line per Wikipedia article.
word-frequencies split \ --input-path $HOME/datasets/wikipedia/plwiki-20200113-cirrussearch-content.json.gz \ --output-dir $HOME/datasets/wikipedia/plwiki-20200113-split
After splitting you can create a frequencies file, which contains counts for unigrams (single words) and bigrams (pairs of words):
word-frequencies create-frequencies \ --input-dir $HOME/datasets/wikipedia/plwiki-20200113-split \ --output-file $HOME/datasets/wikipedia/plwiki-20200113-split/plwiki-20200113-frequencies.txt \ --language pl
This will create a compressed file plwiki-20200113-frequencies.txt.gz
. If you zless
it you can see it contains counts that can
let you build a language model if you'd like.
For now if you only care about the most popular K unigrams, e.g. top 10k words, you can run:
word-frequencies top-k-words \ --number-of-words 10000 \ --input-file $HOME/datasets/wikipedia/plwiki-20200113-split/plwiki-20200113-frequencies.txt.gz \ --output-file $HOME/datasets/wikipedia/plwiki-20200113-split/plwiki-20200113-top-10k.txt
2b. Windows command-line example
TODO, works but need to write out commands and test it
Dictionary sources
English
From https://packages.debian.org/sid/wordlist
download wamerican
, wbritish
, wcanadian
standard lists
(around 103k words each), then concatenate, sort, de-dupe:
cat wamerican/usr/share/dict/american-english \ wbritish/usr/share/dict/british-english \ wcanadian/usr/share/dict/canadian-english | sort | uniq > en.txt
Note that http://wordlist.aspell.net/12dicts-readme/ is another great resource for curated English words.
Polish
The Debian wpolish
dictionary is surprisingly low quality so I scraped Wiktionary to build a Polish dictionary, see below.
Using Wiktionary
I haven't ironed this out but here is some quick Python code to convert Wiktionary dataset dumps (from the same links as above) to dictionary files. You can then put these into the "dictionaries" sub-folder and re-run.
#!/usr/bin/env python3 import json import gzip import unicodedata def main(): main_en() main_pl() def main_pl(): words = [] with gzip.open("plwiktionary-20200113-cirrussearch-content.json.gz", "rb") as f: for line in f: data = json.loads(line) if "language" not in data: continue if " " in data["title"]: continue if "Szablon:język polski" not in data["template"]: continue word = unicodedata.normalize("NFKC", data["title"]) words.append(word) words.sort() with open("pl.txt", "w") as f_out: for word in words: f_out.write("{0}\n".format(word)) def main_en(): words = [] with gzip.open("enwiktionary-20200113-cirrussearch-content.json.gz", "rb") as f: for line in f: data = json.loads(line) if "language" not in data: continue if " " in data["title"]: continue if "English" not in data["heading"]: continue word = unicodedata.normalize("NFKC", data["title"]) words.append(word) words.sort() with open("en.txt", "w") as f_out: for word in words: f_out.write("{0}\n".format(word)) if __name__ == "__main__": main()
TODOs
- Need tests.
- Option to specify your own dictionary file, that way we don't need to keep adding dictionaries to the binary.
-
Very memory inefficient, need ~15GB RAM for English.
- Try interning Strings, I think the string copying is a big culprit.
- If still not good enough then use SQLite to count words.
-
Make minimum article count in
create-frequencies
an input parameter. - Once crate is published update installation instructions.
Testing commands for older English dataset
word-frequencies split \ --input-path $HOME/datasets/wikipedia/enwiki-20191202-cirrussearch-content.json.gz \ --output-dir $HOME/datasets/wikipedia/enwiki-20191202-split word-frequencies create-frequencies \ --input-dir $HOME/datasets/wikipedia/enwiki-20191202-split \ --output-file $HOME/datasets/wikipedia/enwiki-20191202-split/enwiki-20191202-frequencies.txt \ --language en word-frequencies top-k-words \ --number-of-words 10000 \ --input-file $HOME/datasets/wikipedia/enwiki-20191202-split/enwiki-20191202-frequencies.txt.gz \ --output-file $HOME/datasets/wikipedia/enwiki-20191202-split/enwiki-20191202-top-10k.txt echo done
License
word-frequencies
is distributed under the terms of the Apache License (Version 2.0). See LICENSE
for
details.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK