Skip to content

Releases: piskvorky/gensim

3.4.0

01 Mar 10:34
@menshikh-iv menshikh-iv
3.4.0
885430d
Compare
Choose a tag to compare
3.4.0

3.4.0, 2018-03-01

🌟 New features:

👍 Improvements:

  • New method to show the Gensim installation parameters: python -m gensim.scripts.package_info --info. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903)
  • Added a flag to optionally skip network-related tests, to help maintainers avoid network issues with CI services (@menshikh-iv, #1930)
  • Added license field to setup.py, allowing the use of tools like pip-licenses (@nils-werner, #1909)

🔴 Bug fixes:

📚 Tutorial and doc improvements:

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • `gensim.scripts.make_wikicorpus...
Read more
Assets 26

3.3.0

02 Feb 13:50
@menshikh-iv menshikh-iv
3.3.0
1afaacc
Compare
Choose a tag to compare
3.3.0

3.3.0, 2018-02-02

🌟 New features:

  • Re-designed all "*2vec" implementations (@manneshiva, #1777)

  • Improve gensim.scripts.segment_wiki by retaining interwiki links. Fix #1712
    (@steremma, PR #1839)

    • Optionally extract interlinks from Wikipedia pages (use the --include-interlinks option). This will output one additional JSON dict for each article:

      {
    "interlinks": {
        "article title 1": "interlink text 1",
        "article title 2": "interlink text 2",
        ...
    }
}

    • Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump:

      python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz
      • Read this field from the segment_wiki output:
      import json
from smart_open import smart_open

with smart_open("enwiki-latest.jsonl.gz") as infile:
    for doc in infile:
        doc = json.loads(doc)

        src_node = doc['title']
        dst_nodes = doc['interlinks'].keys()

        print(u"Source node: {}".format(src_node))
        print(u"Destination nodes: {}".format(u", ".join(dst_nodes)))
        break

"""
OUTPUT:

Source node: Anarchism
Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, A...
Read more
Assets 26

Christmas Come Early

09 Dec 19:19
@menshikh-iv menshikh-iv
3.2.0
6dd8ae7
Compare
Choose a tag to compare
Christmas Come Early

3.2.0, 2017-12-09

🌟 New features:

  • New download API for corpora and pre-trained models (@chaitaliSaini & @menshikh-iv, #1705 & #1632 & #1492)

    • Download large NLP datasets in one line of Python, then use with memory-efficient data streaming: 
      import gensim.downloader as api

for article in api.load("wiki-english-20171001"):
    print(article)
    • Don’t waste time searching for good word embeddings, use the curated ones: 
      import gensim.downloader as api

model = api.load("glove-twitter-25")
model.most_similar("engineer")

# [('specialist', 0.957542896270752),
#  ('developer', 0.9548177123069763),
#  ('administrator', 0.9432312846183777),
#  ('consultant', 0.93915855884552),
#  ('technician', 0.9368376135826111),
#  ('analyst', 0.9342101216316223),
#  ('architect', 0.9257484674453735),
#  ('engineering', 0.9159940481185913),
#  ('systems', 0.9123805165290833),
#  ('consulting', 0.9112802147865295)]
    • Blog post introducing the API and design decisions.
    • Jupyter notebook with examples

  • New model: Poincaré embeddings (@jayantj, #1696 & #1700 & #1757 & #1734)

    • Embed a graph (taxonomy) in the same way as word2vec embeds words: 
      from gensim.models.poincare import PoincareRelations, PoincareModel
from gensim.test.utils import datapath

data = PoincareRelations(datapath('poincare_hypernyms.tsv'))
model = PoincareModel(data)
model.kv.most_similar("cat.n.01")

# [('kangaroo.n.01', 0.010581353439700418),
# ('gib.n.02', 0.011171531439892076),
# ('striped_skunk.n.01', 0.012025106076442395),
# ('metatherian.n.01', 0.01246679759214648),
# ('mammal.n.01', 0.013281303506525968),
# ('marsupial.n.01', 0.013941330203709653)]
    • Tutorial on Poincaré embeddings (Jupyter notebook).
    • Model introduction and the journey of its implementation (blog post).
    • Original paper on arXiv.

  • Optimized FastText (@manneshiva, #1742)

    • New fast multithreaded implementation of FastText, natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation. 
      import gensim.downloader as api
from gensim.models import FastText

model = FastText(api.load("text8"))
model.most_similar("cat")

# [('catnip', 0.8538144826889038),
#  ('catwalk', 0.8136177062988281),
#  ('catchy', 0.7828493118286133),
#  ('caf', 0.7826495170593262),
#  ('bobcat', 0.7745151519775391),
#  ('tomcat', 0.7732658386230469),
#  ('moat', 0.7728310823440552),
#  ('caye', 0.7666271328926086),
#  ('catv', 0.7651021480560303),
#  ('caveat', 0.7643581628799438)]

  • Binary pre-compiled wheels for Windows, OSX and Linux (@menshikh-iv, MacPython/gensim-wheels/#7)

    • Users no longer need to have a C compiler for using the fast (Cythonized) version of word2vec, doc2vec, fasttext etc.
    • Faster Gensim pip installation

  • Added DeprecationWarnings to deprecated methods and parameters, with a clear schedule for removal.

👍 Improvements:

🔴 Bug fixes:

📚 Tutorial and doc improvements:

  • Update perf numbers of segment_wiki (...
Read more
Assets 26

3.1.0

06 Nov 16:56
@menshikh-iv menshikh-iv
3.1.0
b6234e7
Compare
Choose a tag to compare
3.1.0

3.1.0, 2017-11-06

🌟 New features:

  • Massive optimizations to LSI model training (@isamaru, #1620 & #1622)

    • LSI model allows use of single precision (float32), to consume 40% less memory while being 40% faster.

    • LSI model can now also accept CSC matrix as input, for further memory and speed boost.

    • Overall, if your entire corpus fits in RAM: 3x faster LSI training (SVD) in 4x less memory!

      # just an example; the corpus stream is up to you
streaming_corpus = gensim.corpora.MmCorpus("my_tfidf_corpus.mm.gz")

# convert your corpus to a CSC sparse matrix (assumes the entire corpus fits in RAM)
in_memory_csc_matrix = gensim.matutils.corpus2csc(streaming_corpus, dtype=np.float32)

# then pass the CSC to LsiModel directly
model = LsiModel(corpus=in_memory_csc_matrix, num_topics=500, dtype=np.float32)

    • Even if you continue to use streaming corpora (your training dataset is too large for RAM), you should see significantly faster processing times and a lower memory footprint. In our experiments with a very large LSI model, we saw a drop from 29 GB peak RAM and 38 minutes (before) to 19 GB peak RAM and 26 minutes (now):

      model = LsiModel(corpus=streaming_corpus, num_topics=500, dtype=np.float32)

  • Add common terms to Phrases. Fix #1258 (@alexgarel, #1568)

    • Phrases allows to use common terms in bigrams. Before, if you are searching to reveal ngrams like car_with_driver and car_without_driver, you can either remove stop words before processing, but you will only find car_driver, or you won't find any of those forms (because they have three words, but also because high frequency of with will avoid them to be scored correctly), inspired by ES common grams token filter.

      phr_old = Phrases(corpus)
phr_new = Phrases(corpus, common_terms=stopwords.words('en'))

print(phr_old[["we", "provide", "car", "with", "driver"]])  # ["we", "provide", "car_with", "driver"]
print(phr_new[["we", "provide", "car", "with", "driver"]])  # ["we", "provide", "car_with_driver"]

  • New segment_wiki.py script (@menshikh-iv, #1483 & #1694)

    • CLI script for processing a raw Wikipedia dump (the xml.bz2 format provided by MediaWiki) to extract its articles in a plain text format. It extracts each article's title, section names and section content and saves them as json-line:

      python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 | gzip > enwiki-latest-pages-articles.json.gz

      Processing the entire English Wikipedia dump (13.5 GB, link here) takes about 2.5 hours (i7-6700HQ, SSD).

      The output format is one article per line, serialized into JSON:

       for line in smart_open('enwiki-latest-pages-articles.json.gz'):  # read the file we just created
     article = json.loads(line)
     print("Article title: %s" % article['title'])
     for section_title, section_text in zip(article['section_titles'], article['section_texts']):
         print("Section title: %s" % section_title)
         print("Section text: %s" % section_text)

👍 Improvements:

🔴 Bug fixes:

📚 Tutorial and doc improvements:

⚠️ Deprecation part (will come into force in the next major release)

  • Remove

    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next major release.

Assets 2

3.0.1

12 Oct 10:11
@menshikh-iv menshikh-iv
3.0.1
86e0618
Compare
Choose a tag to compare
3.0.1

3.0.1, 2017-10-12

🔴 Bug fixes:

📚 Tutorial and doc improvements:

⚠️ Deprecation part (will come into force in the next release)

  • Remove

    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next release.

Assets 2

GSoC storm

27 Sep 09:00
@menshikh-iv menshikh-iv
3.0.0
351bdef
Compare
Choose a tag to compare
GSoC storm

3.0.0, 2017-09-27

🌟 New features:

👍 Improvements:

🔴 Bug fixes:

📚 Tutorial and doc improvements:

Assets 2

Docker image and integration with Sklearn

25 Jul 14:11
@menshikh-iv menshikh-iv
2.3.0
2cdf685
Compare
Choose a tag to compare
Docker image and integration with Sklearn

2.3.0, 2017-07-25

🌟 New features:

👍 Improvements:

🔴 Bug fixes:

📚 Tutorial and doc improvements:

Assets 2

Integration with Keras and Sklearn, LdaModel topic difference

21 Jun 17:05
@menshikh-iv menshikh-iv
2.2.0
a3965e9
Compare
Choose a tag to compare
Integration with Keras and Sklearn, LdaModel topic difference

2.2.0, 2017-06-21

🌟 New features:

👍 Improvements:

🔴 Bug fixes:

📚 Tutorial and doc improvements:

Assets 2

Doc2Vec visualisation

12 May 11:49
@menshikh-iv menshikh-iv
2.1.0
1b8fdaf
Compare
Choose a tag to compare
Doc2Vec visualisation

2.1.0, 2017-05-12

🌟 New features:

  • Add modified save_word2vec_format for Doc2Vec, to save document vectors. (@parulsethi, #1256)

👍 Improvements:

  • Add automatic code style check limited only to the code modified in PR (@tmylk, #1287)
  • Replace logger.warn by logger.warning (@chinmayapancholi13, #1295)
  • Docs word2vec docstring improvement, deprecation labels (@shubhvachher, #1274)
  • Stop passing 'sentences' as parameter to Doc2Vec. Fix #511 (@gogokaradjov, #1306)

🔴 Bug fixes:

📚 Tutorial and doc improvements:

Assets 2

2.0.0, 2017-04-10

10 Apr 23:45
@tmylk tmylk
2.0.0
5031f8b
Compare
Choose a tag to compare
2.0.0, 2017-04-10

Breaking changes:

Any direct calls to method train() of Word2Vec/Doc2Vec now require an explicit epochs parameter and explicit estimate of corpus size. The most usual way to call train is vec_model.train(sentences, total_examples=self.corpus_count, epochs=self.iter)
See the method documentation for more information.

New features:

Improvements:

Bug fixes:

Tutorial and doc improvements:

Assets 2