Remove "-" before count of posts on Topic Cards when sorting by "# of People"

[ebarry]

Please describe the problem

There appears to be a tiny "-" before the count of posts on Topic Cards when the "# of People" sorting option is clicked, making it appear like there is a negative count of posts. See screenshot:

What did you expect to see that you didn't?

i expected to see positive integers.

Please show us where to look

https://publiclab.org/tags?sort=followers&order=asc

What's your PublicLab.org username?

This can help us diagnose the issue:

liz

[ebarry]

Does anyone have code links at hand that they could add here? Thanks in advance!

[Tlazypanda]

Hey @ebarry I think this is a more serious issue actually 😅 I had made changes for this here #7609 but I must have messed it up and calculated the wrong count I think 😞 Looping in @jywarren Sorry might need some help to fix this up 😅

[jywarren]

Hi @Tlazypanda can we look at other places where Tag.counter type code is used, and try to switch to that?

https://github.com/publiclab/plots2/pull/7609/files#diff-8fe92870a7f027a46eed82a51c5c16bcR118

However, the counts are off by very little. I wonder if this is an issue of some things being spammed and then that affecting the math a little bit?

See for example the difference between these counts:

We can look for where the 236 count is calculated and switch over to that, perhaps?

So that uses a controller-generated instance variable:

plots2/app/views/tag/show.html.erb

Line 58 in c056cb0

<p class="contributor-info"><%= @total_posts %> posts by

I believe this is where this is calculated:

plots2/app/controllers/tag_controller.rb

Lines 551 to 560 in c056cb0

	@total_posts = case @node_type
	when 'note'
	@notes.count
	when 'questions'
	@questions.count
	when 'wiki'
	@wikis.count
	when 'maps'
	@nodes.count
	end

[urvashigupta7]

Hey @jywarren, what I can see in tag/:id i.e. the showpage for tag is that we are displaying only research notes over there and in topic cards , we are trying to display research notes+questions.
So, I think what we need to do is that we need to fetch notes in the same way as we are doing in tag#show and for displaying the count at the bottom of topic card we will just need to use this math expression:
Number of notes with the tagname-Count of displayed notes in Topic Card

[jywarren]

we need to fetch notes in the same way as we are doing in tag#show

Hi, that's just right! noting the concept of a "single source of truth" here: #6855 (comment)

I believe we should be using the same calculations here: #7476 (based on #6855).

@urvashigupta7 thank you so much for chiming in! Would you be interested in making these changes?

[jywarren]

I also think that in #7476, we are not filtering out spam. So on these lines, we should probably add .where(status: 1):

plots2/app/controllers/tag_controller.rb

Lines 551 to 560 in c056cb0

	@total_posts = case @node_type
	when 'note'
	@notes.count
	when 'questions'
	@questions.count
	when 'wiki'
	@wikis.count
	when 'maps'
	@nodes.count
	end

That code also depends on this, which doesn't filter spam:

plots2/app/models/node.rb

Lines 609 to 618 in f4dfdcb

	def self.for_tagname_and_type(tagname, type = 'note', options = {})
	return Node.for_wildcard_tagname_and_type(tagname, type) if options[:wildcard]
	return Node.for_question_tagname_and_type(tagname, type) if options[:question]
	Node.where(status: 1, type: type)
	.includes(:revision, :tag)
	.references(:term_data, :node_revisions)
	.where('term_data.name = ? OR term_data.parent = ?', tagname, tagname)
	end

Likewise this code needs also to filter spam out with the same filter:

plots2/app/models/tag.rb

Lines 118 to 124 in f4dfdcb

	def self.counter(tagname)
	Node.where(type: %w(note page))
	.where('term_data.name = ?', tagname)
	.includes(:node_tag, :tag)
	.references(:term_data)
	.count
	end

[jywarren]

Hmm. It does seem like we are filtering spam from the @total_posts calculation, as it runs /after/ these lines:

plots2/app/controllers/tag_controller.rb

Lines 204 to 218 in c056cb0

	nodes = Tag.tagged_nodes_by_author(@tagname, @user)
	.where(status: 1, type: node_type)
	@total_posts = nodes.count
	nodes = nodes.paginate(page: params[:page], per_page: 24)
	@notes ||= []
	@notes = nodes.where('node.nid NOT IN (?)', qids) if @node_type == 'note'
	@questions = nodes.where('node.nid IN (?)', qids) if @node_type == 'questions'
	ans_ques = Answer.where(uid: @user.id, accepted: true).includes(:node).map(&:node)
	@answered_questions = ans_ques.paginate(page: params[:page], per_page: 24)
	@wikis = nodes if @node_type == 'wiki'
	@nodes = nodes if @node_type == 'maps'
	@title = "'" + @tagname.to_s + "' by " + params[:author]

However, neither Tag.counter and Node.for_tagname_and_type filter. Maybe that is the irregularity?

[jywarren]

The count here is still not right, i think. For example https://stable.publiclab.org/tags now lists 692 posts for balloon-mapping but here are the stats for that tag (in stable):

I think we need to depend on this fix: #8245

then we can check this again. Reopening for a moment... fingers 🤞

[jywarren]

#8245 is now done! And has unit tests, so nice. OK, now this may work...

[jywarren]

OK, it says 573 posts » -- hmm

Research notes 469
Questions 37
Wiki pages 77
People 346

I mean, notes plus wikis = 546...? and plus questions is 583?

[jywarren]

Now, testing the tag tests on stable, I'm suspicious that there may be some caching going on. Adding that tag to a new note, https://stable.publiclab.org/notes/aidanswope/05-21-2019/angel-s-point-balloon-map-updated, did not result in the tab of notes going from 7 to 8, although tag.count returns 8 as does the result of tag.run_count

https://stable.publiclab.org/tag/tests

[jywarren]

and noting link to #8244

[jywarren]

YESSSSS it's 5-minute cached:

plots2/app/views/tag/show.html.erb

Line 133 in e97eff7

<% cache("tag-show-#{request.fullpath}", expires_in: 5.minutes, skip_digest: true) do %>

[jywarren]

OK, so tracing the tab count:

plots2/app/models/node.rb

Lines 621 to 630 in e97eff7

	def self.for_tagname_and_type(tagname, type = 'note', options = {})
	return Node.for_wildcard_tagname_and_type(tagname, type) if options[:wildcard]
	return Node.for_question_tagname_and_type(tagname, type) if options[:question]
	Node.where(status: 1, type: type)
	.includes(:revision, :tag)
	.references(:term_data, :node_revisions)
	.where('term_data.name = ? OR term_data.parent = ?', tagname, tagname)
	end

Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').where.not(nid: qids).count => 470

Versus the topic card count:

NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').where.not(nid: qids).count => 467

😭

[jywarren]

These three are the difference: [6129, 7236, 7414] --

[#<Node nid: 6129, vid: 11131, type: "note", language: "en", title: "Balloon Mapping Test Flight - Riverside Cemetary, ...", uid: 51650, status: 1, created: 1361913526, changed: 1484959040, comment: 2, promote: 1, moderate: 0, sticky: 0, tnid: 0, translate: 0, cached_likes: 0, comments_count: 1, drupal_node_revisions_count: 1, path: "/notes/christenmcnamara/2-26-2013/balloon-mapping-...", main_image_id: nil, slug: "christenmcnamara-02-26-2013-balloon-mapping-test-f...", views: 148, latitude: nil, longitude: nil, precision: nil, flag: 0>,
#<Node nid: 7236, vid: 12787, type: "note", language: "en", title: "Fresh Kills Balloon Mapping", uid: 53894, status: 1, created: 1367679102, changed: 1484959046, comment: 2, promote: 1, moderate: 0, sticky: 0, tnid: 0, translate: 0, cached_likes: 1, comments_count: 2, drupal_node_revisions_count: 1, path: "/notes/nejohnson2/5-4-2013/fresh-kills-balloon-map...", main_image_id: nil, slug: "nicholas-05-04-2013-fresh-kills-balloon-mapping", views: 165, latitude: nil, longitude: nil, precision: nil, flag: 0>,
#<Node nid: 7414, vid: 13006, type: "note", language: "en", title: "Help requested with mapknitter- Balloon mapping of...", uid: 46536, status: 1, created: 1368460800, changed: 1484959048, comment: 2, promote: 1, moderate: 0, sticky: 0, tnid: 0, translate: 0, cached_likes: 1, comments_count: 4, drupal_node_revisions_count: 1, path: "/notes/lynnrw/5-10-2013/help-requested-mapknitter-...", main_image_id: nil, slug: "lynnrw-05-13-2013-help-requested-with-mapknitter-b...", views: 613, latitude: nil, longitude: nil, precision: nil, flag: 0>]

[jywarren]

VERY strange! for 2 reasons: 3 notes is not the same as the 10 missing ones (thought we haven't looked at questions yet), AND, these three seem to be normal except that one was in Feb 2013 and 2 were in May 2013:

• https://stable.publiclab.org/n/6129
• https://stable.publiclab.org/n/7236
• https://stable.publiclab.org/n/7414

(update: see below for duplicate tagnames)

[jywarren]

Also interesting:

Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').where(nid: qids).count => 29

This agrees with:

NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').where(nid: qids).count

vs. 37 as shown on the tab, which is calculated with:

plots2/app/models/node.rb

Lines 640 to 646 in e97eff7

	def self.for_question_tagname_and_type(tagname, type = 'note')
	other_tag = tagname.include?("question:") ? tagname.split(':')[1] : "question:#{tagname}"
	Node.where(status: 1, type: type)
	.includes(:revision, :tag)
	.references(:term_data, :node_revisions)
	.where('term_data.name = ? OR term_data.name = ? OR term_data.parent = ?', tagname, other_tag, tagname)
	end

[jywarren]

Aha! That last section shows that posts tagged question:balloon-mapping are included in that count even if they do not have the base balloon-mapping tag!!!

[jywarren]

The following 8 posts seem to have question:balloon-mapping but not balloon-mapping:

[13556, 15762, 16426, 18994, 20694, 11301, 12251, 19625]

[jywarren]

OK, so that gets us to within 2 nodes. Now let's just manually compare:

irb(main):040:0> NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').count
=> 496
irb(main):041:0> Node.where(status: 1, type: 'note').includes(:tag).references(:term_data).where('term_data.name = ?', 'balloon-mapping').count
=> 499

irb(main):022:0> Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').collect(&:nid)
=> [146, 253, 254, 268, 403, 511, 724, 732, 743, 913, 1482, 1489, 1620, 1626, 1627, 1632, 1715, 1849, 1968, 1994, 2126, 2138, 2144, 2476, 2589, 2930, 3115, 3325, 4649, 4952, 5038, 5220, 5250, 5978, 6934, 7050, 7189, 7294, 7314, 7491, 7577, 7593, 7995, 8189, 8907, 8913, 8914, 8950, 9019, 9032, 9042, 9086, 9417, 9434, 9742, 9888, 9977, 10541, 11019, 11100, 11223, 11574, 11582, 11827, 12201, 12240, 12491, 12619, 12708, 13095, 13424, 13945, 14022, 14095, 14183, 14185, 14219, 14515, 14572, 14587, 14597, 14600, 14611, 14620, 14813, 14907, 14909, 14983, 15545, 15568, 16591, 16653, 16746, 16877, 18316, 19153, 19272, 19462, 19558, 19578, 19620, 19762, 20331, 23549, 24101, 24102, 1795, 6635, 347, 369, 439, 444, 792, 861, 1665, 1668, 1962, 2135, 2143, 2152, 2245, 2277, 2783, 2784, 3563, 3635, 3670, 4482, 4516, 4755, 4760, 4923, 4925, 6059, 7641, 10181, 10489, 11101, 11724, 13243, 13880, 13948, 17322, 64, 166, 202, 344, 1666, 1816, 1818, 1819, 2919, 3354, 3736, 3821, 4087, 8020, 8947, 9301, 11199, 11260, 11261, 11755, 12161, 1544, 1739, 1742, 1892, 2006, 5494, 9353, 10705, 828, 1515, 1786, 1877, 1926, 1949, 2121, 2625, 2666, 2677, 2846, 2892, 2961, 4092, 4907, 5394, 5415, 5664, 7132, 7367, 7368, 7717, 7964, 8400, 10243, 6175, 9507, 4636, 6293, 3014, 6624, 9806, 10289, 10554, 11278, 11391, 12104, 12622, 12894, 13143, 9935, 10036, 10196, 10450, 11283, 11719, 11769, 12237, 10246, 10605, 2462, 9497, 1889, 2131, 2443, 3026, 4127, 4252, 4673, 5538, 5893, 6608, 8034, 8902, 8959, 9644, 9707, 10045, 10162, 10335, 10336, 10337, 10373, 10540, 10570, 11107, 11110, 11112, 11124, 11143, 11175, 11298, 11336, 12053, 12203, 12590, 16984, 19515, 19580, 23822, 2163, 3634, 7620, 10166, 11056, 11319, 11589, 1541, 1652, 1676, 1677, 1746, 2154, 2237, 2275, 2311, 2623, 2785, 3454, 3652, 3822, 4014, 4691, 4720, 4950, 5161, 7785, 8866, 8926, 8928, 8937, 8958, 8986, 9759, 9915, 10101, 10230, 10244, 10302, 10311, 10374, 10523, 10602, 11021, 11078, 11103, 11253, 11726, 11988, 12686, 12752, 12757, 13400, 13845, 14116, 14660, 14713, 14756, 1716, 2660, 7191, 7192, 1743, 1636, 1900, 9317, 10651, 1741, 13904, 3096, 2352, 2824, 4091, 2715, 6080, 6082, 2732, 3801, 3908, 4545, 7121, 10284, 13791, 10631, 4921, 6751, 9429, 9743, 11577, 17331, 6073, 9858, 10711, 11118, 6525, 10267, 9034, 11252, 11268, 8033, 8868, 10159, 10214, 9027, 9785, 9033, 9076, 9299, 14501, 14508, 14562, 21733, 10282, 9909, 11178, 11738, 10024, 12667, 11262, 11343, 11978, 11983, 11992, 11826, 11061, 10227, 10428, 11077, 11131, 11137, 11280, 11281, 11518, 11551, 11807, 11812, 11947, 12039, 14158, 14729, 14943, 15641, 16273, 16349, 18560, 19269, 19286, 21547, 10343, 14852, 14901, 15352, 15387, 10612, 10638, 10986, 10994, 20181, 11002, 11127, 11128, 11183, 12166, 11342, 11494, 11955, 11673, 11707, 11711, 12190, 18839, 13802, 14595, 12227, 15912, 14007, 13093, 13096, 12981, 13102, 13160, 13194, 13196, 13227, 13242, 13267, 13278, 13281, 13298, 14168, 15161, 23143, 14058, 14766, 13748, 13839, 15603, 14241, 14179, 14239, 14246, 14240, 14255, 14488, 14487, 14245, 14785, 15160, 14703, 14995, 16571, 15172, 15179, 15424, 15713, 15732, 16141, 16284, 16334, 16382, 17387, 17498, 17499, 18498, 16960, 19689, 17815, 19394, 19464, 19650, 19955, 19476, 19477, 23203, 6129, 7236, 7414]

vs:

irb(main):024:0> NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').collect(&:nid)
=> [146, 253, 254, 268, 403, 511, 724, 732, 743, 913, 1482, 1489, 1620, 1626, 1627, 1632, 1715, 1849, 1968, 1994, 2126, 2138, 2144, 2476, 2589, 2930, 3115, 3325, 4649, 4952, 5038, 5220, 5250, 5978, 6934, 7050, 7189, 7294, 7314, 7491, 7577, 7593, 7995, 8189, 8907, 8913, 8914, 8950, 9019, 9032, 9042, 9086, 9417, 9434, 9742, 9888, 9977, 10541, 11019, 11100, 11223, 11574, 11582, 11827, 12201, 12240, 12491, 12619, 12708, 13095, 13424, 13945, 14022, 14095, 14183, 14185, 14219, 14515, 14572, 14587, 14597, 14600, 14611, 14620, 14813, 14907, 14909, 14983, 15545, 15568, 16591, 16653, 16746, 16877, 18316, 19153, 19272, 19462, 19558, 19578, 19620, 19762, 20331, 23549, 24101, 24102, 1795, 6635, 347, 369, 439, 444, 792, 861, 1665, 1668, 1962, 2135, 2143, 2152, 2245, 2277, 2783, 2784, 3563, 3635, 3670, 4482, 4516, 4755, 4760, 4923, 4925, 6059, 7641, 10181, 10489, 11101, 11724, 13243, 13880, 13948, 17322, 64, 166, 202, 344, 1666, 1816, 1818, 1819, 2919, 3354, 3736, 3821, 4087, 8020, 8947, 9301, 11199, 11260, 11261, 11755, 12161, 1544, 1739, 1742, 1892, 2006, 5494, 9353, 10705, 828, 1515, 1786, 1877, 1926, 1949, 2121, 2625, 2666, 2677, 2846, 2892, 2961, 4092, 4907, 5394, 5415, 5664, 7132, 7367, 7368, 7717, 7964, 8400, 10243, 6175, 9507, 4636, 6293, 3014, 6624, 9806, 10289, 10554, 11278, 11391, 12104, 12622, 12894, 13143, 9935, 10036, 10196, 10450, 11283, 11719, 11769, 12237, 10246, 10605, 2462, 9497, 1889, 2131, 2443, 3026, 4127, 4252, 4673, 5538, 5893, 6608, 8034, 8902, 8959, 9644, 9707, 10045, 10162, 10335, 10336, 10337, 10373, 10540, 10570, 11107, 11110, 11112, 11124, 11143, 11175, 11298, 11336, 12053, 12203, 12590, 16984, 19515, 19580, 23822, 2163, 3634, 7620, 10166, 11056, 11319, 11589, 1541, 1652, 1676, 1677, 1746, 2154, 2237, 2275, 2311, 2623, 2785, 3454, 3652, 3822, 4014, 4691, 4720, 4950, 5161, 7785, 8866, 8926, 8928, 8937, 8958, 8986, 9759, 9915, 10101, 10230, 10244, 10302, 10311, 10374, 10523, 10602, 11021, 11078, 11103, 11253, 11726, 11988, 12686, 12752, 12757, 13400, 13845, 14116, 14660, 14713, 14756, 1716, 2660, 7191, 7192, 1743, 1636, 1900, 9317, 10651, 1741, 13904, 3096, 2352, 2824, 4091, 2715, 6080, 6082, 2732, 3801, 3908, 4545, 7121, 10284, 13791, 10631, 4921, 6751, 9429, 9743, 11577, 17331, 6073, 9858, 10711, 11118, 6525, 10267, 9034, 11252, 11268, 8033, 8868, 10159, 10214, 9027, 9785, 9033, 9076, 9299, 14501, 14508, 14562, 21733, 10282, 9909, 11178, 11738, 10024, 12667, 11262, 11343, 11978, 11983, 11992, 11826, 11061, 10227, 10428, 11077, 11131, 11137, 11280, 11281, 11518, 11551, 11807, 11812, 11947, 12039, 14158, 14729, 14943, 15641, 16273, 16349, 18560, 19269, 19286, 21547, 10343, 14852, 14901, 15352, 15387, 10612, 10638, 10986, 10994, 20181, 11002, 11127, 11128, 11183, 12166, 11342, 11494, 11955, 11673, 11707, 11711, 12190, 18839, 13802, 14595, 12227, 15912, 14007, 13093, 13096, 12981, 13102, 13160, 13194, 13196, 13227, 13242, 13267, 13278, 13281, 13298, 14168, 15161, 23143, 14058, 14766, 13748, 13839, 15603, 14241, 14179, 14239, 14246, 14240, 14255, 14488, 14487, 14245, 14785, 15160, 14703, 14995, 16571, 15172, 15179, 15424, 15713, 15732, 16141, 16284, 16334, 16382, 17387, 17498, 17499, 18498, 16960, 19689, 17815, 19394, 19464, 19650, 19955, 19476, 19477, 23203]

[jywarren]

Aha. Maybe this:

Tag.where(name: 'balloon-mapping').length => 4
Tag.where(name: 'balloon-mapping').collect(&:tid) => [10, 3205, 3591, 4032]

[jywarren]

Yes, that's the discrepancy:

irb(main):045:0> NodeTag.joins(:node).where(tid: [10, 3205, 3591, 4032]).where('node.status = 1 AND node.type = "note"').count
=> 499
irb(main):046:0> Node.where(status: 1, type: 'note').includes(:tag).references(:term_data).where('term_data.name = ?', 'balloon-mapping').count
=> 499

irb(main):050:0> Tag.where(name: 'balloon-mapping').collect(&:count)
=> [574, 1, 1, 1]

[jywarren]

However, notably not for the method or project bugs in #7334

irb(main):047:0> Tag.where(name: 'method').collect(&:tid)
=> [14474]
irb(main):048:0> Tag.where(name: 'project').collect(&:tid)
=> [14751]

[jywarren]

OK, i fixed the duplicate tags with:

irb(main):053:0> NodeTag.where(tid: [3205, 3591, 4032]).each do |nt|
irb(main):054:1* nt.tid = 10
irb(main):055:1> nt.save
irb(main):056:1> end

however i bet there are more in the DB. We may want to do a bigger project to consolidate these.

Update: yes, there are 28 other instances:

irb(main):063:0> Tag.count
=> 18532
irb(main):065:0> Tag.all.collect(&:name).uniq.count
=> 18504


irb(main):069:0> (Tag.all - Tag.all.group(:name)).collect(&:name)
=> ["fold-up", "gulf-coast", "kite-mapping", "gulf-coast", "brooklyn", "new-york-city", "balloon-mapping", "spectrometer", "farmhack", "white-balance", "fresh-kills", "balloon-mapping", "balloon-mapping", "kite-mapping", "newsletter", "newsletter", "passenger-pigeon", "jamaica-bay", "mobile-spectrometer", "infragram-sandbox", "mailing-lists", "diy-kite", "water-sensing", "carbon-arc", "diy-kite", "diy-kite", "workplace", "emergencies"]

[jywarren]

My recommendation now is:

We've narrowed this down a lot, and it has two parts.

1. “ghost” tags - duplicate tagnames so basically some weren’t being counted (solved, although ~15 other instances exist for a few other tags)
2. question:_____ format tags, which are counted under the questions tab on /tag/____ pages, but if they don’t have their own base tag, are not counted in the topic cards. Example: 8 posts have question:balloon-mapping but not balloon-mapping , so they throw our count by 8.

My take on (2) is that we should write it down and be aware of it, but not solve it (i can’t think of a clean solution anyways).

And, even after all that digging, I still have a discrepancy of 1 for balloon-mapping. But I think we are so close that we should essentially be OK with that, as it’s one of the largest tags and that issue is unlikely to affect others. Interested to hear others' take on that too!