Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does anything other than datagrepper use the datanommer models API directly #151

Open
ryanlerch opened this issue Jul 15, 2021 · 9 comments
Open

Does anything other than datagrepper use the datanommer models API directly #151

ryanlerch opened this issue Jul 15, 2021 · 9 comments
Assignees
@ryanlerch
Labels
L Large Task (about 15 hours)
Projects
Datanommer / Datagrepper

Comments

@ryanlerch
Copy link
Contributor

The datanommer readme states that the majority of tools and applications interact with the datanommer db using datagrepper, with the exception of datagrepper itself, and fedbadges.

The questions here are:

  • does fedbadges still connect directly to the datanommer db?
  • does anything else connect directly to the datanommer db?
@ryanlerch ryanlerch added this to Backlog in Datanommer / Datagrepper via automation Jul 15, 2021
@abompard
Copy link
Member

Yeah, the badges do connect to the datanommer db directly using the models. We'll need to update this piece of code too.

@abompard abompard moved this from Backlog to To do in Datanommer / Datagrepper Aug 11, 2021
@abompard abompard added the L Large Task (about 15 hours) label Aug 11, 2021
@ryanlerch
Copy link
Contributor Author

So did a little digging into this, and it appears that FMN also uses datanommer.models, but these are the only two i could find that use the database models directly. Both these applications seem to only use the grep method in datanommer.models, which is a good thing, as the API for this method has not changed, so we shouldnt need to change code in badges or fmn. We likely will need to update these applications to use the newest version of datanommer.models which shouldnt be a big issue.

Notes on where these two applications use datanommer.models are detailed below:

Fedora Badges

It appears that Fedora Badges is using the datanommer.models "grep" function in fedora-infra/fedbadges , which is the consumer that reads the message queue and awards the badges.

AFAICT, datanommer.models is initted here:

and the grep method called in these two spots:

FMN

datanommer.models is initted here:

and the grep method used once here:

@ryanlerch
Copy link
Contributor Author

OK,

Working on getting badges working in staging to see if we can test this.

@ryanlerch ryanlerch moved this from To do to In progress in Datanommer / Datagrepper Aug 18, 2021
@ryanlerch ryanlerch self-assigned this Aug 18, 2021
@ryanlerch
Copy link
Contributor Author

Ok, so badges is running in staging -- kinda.

The frontend is working -- but the backend fails awarding badges as it appears to stil interface with FAS2 (and there is no FAS2 for staging... so...)

Anyway, long story short, we can still test out the new datanommer by changing the url on staging badges-backend (/etc/fedmsg.d/datanommer.py) to be:

config = {
    'datanommer.sqlalchemy.url': 'postgresql://datanommer:ohZoad7Foowe2the4Zae5ovee8Iquaxi@db-datanommer01.stg.iad2.fedoraproject.org/datanommer2',
}

and we get this error:

Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: 'SELECT count(*) AS count_1 \nFROM (SELECT messages.id AS messages_id, messages.msg_id AS messages_msg_id, messages.i AS messages_i, messages.topic AS messages
_topic, messages.timestamp AS messages_timestamp, messages.certificate AS messages_certificate, messages.signature AS messages_signature, messages.category AS messages_category, messages.username AS messages_username, messages.crypto AS m
essages_crypto, messages.source_name AS messages_source_name, messages.source_version AS messages_source_version, messages._msg AS messages__msg, messages._headers AS messages__headers \nFROM messages \nWHERE (EXISTS (SELECT 1 \nFROM user_messages, "user" \nWHERE messages.id = user_messages.msg AND "user".name = user_messages.username AND "user".name = %(name_1)s)) AND messages.topic = %(topic_1)s) AS anon_1' {'name_1': u'ryanlerch', 'topic_1': u'org.fedoraproject.stg.wiki.article.edit'}                                                                                                                                                                                                                              Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: Traceback (most recent call last):                                                                                                                             Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 89, in inner                                                                                  
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return method(self, *args, **kwargs)                                                                                                                           Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 385, in matches                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return self.specialization.matches(msg)                                                                                                                        Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 498, in matches                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: total, pages, query = self._construct_query(msg)                                                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 472, in _construct_query                                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: total, pages, query = datanommer.models.Message.grep(**kwargs)                                                                                                 Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/datanommer/models/__init__.py", line 431, in grep                                                                       Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: total = query.count()                                                                                                                                          Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2640, in count                                                                         Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return self.from_self(col).scalar()                                                                                                                            Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2426, in scalar                                                                        Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ret = self.one()                                                                                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2395, in one                                                                           Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ret = list(self)                                                                                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__                                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return self._execute_and_instances(context)                                                                                                                    Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances                                                        
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: result = conn.execute(querycontext.statement, self._params)                                                                                                    Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute                                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return meth(self, multiparams, params)                                                                                                                         Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return connection._execute_clauseelement(self, multiparams, params)                                                                                            Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement                                                       Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: compiled_sql, distilled_params                                                                                                                                 Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context                                                             Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: context)                                                                                                                                                       Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception                                                     
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: exc_info                                                                                                                                                       
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause                                                             
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: reraise(type(exception), exception, tb=exc_tb)                                                                                                                 
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context                                                             
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: context)                                                                                                                                                       
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: cursor.execute(statement, parameters)                                                                                                                          
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ProgrammingError: (ProgrammingError) column messages._msg does not exist                
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: LINE 2: ...ssages.source_version AS messages_source_version, messages._...                                                                                     
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ^                                                                                                                                                              
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: HINT:  Perhaps you meant to reference the column "messages.msg".

Which i guess is because we are running python2.7 on badges backend, and using python-datanommer-models 0.9.0 -- https://src.fedoraproject.org/rpms/python-datanommer-models (this is pulled in as a dep of python-fedbadges, which is the fedmsg backend consumer)

@ryanlerch
Copy link
Contributor Author

However, this is where i stopped, as not sure what the best approach here is -- the two options i see are:

  1. Somehow provide a python 2.7 version of datanommer.models -- not sure this is even possible with the timescaledb stuff we have added.
  2. Update the entire badges stack to python3 (pretty sure given our time contraints, this is out of the picture too)

@abompard
Copy link
Member

Another option would be to switch badges to use HTTP requests instead of using the models directly.

@mscherer
Copy link

Fedbadges (in the develop branch) was ported to python 3: fedora-infra/fedbadges@a3f2b03

However, I think no one really know how to do a release (especially since the CI was not working), and the plan was to just run it in a pod, with the HEAD version (since doing release for a custom software where we are the only user seems a bit overkill)

I am not sure there is no need to port the whole stack, just fedbadges.

@ryanlerch ryanlerch moved this from In progress to Blocked in Datanommer / Datagrepper Aug 25, 2021
@mscherer mscherer mentioned this issue Aug 26, 2021
Open
@ryanlerch
Copy link
Contributor Author

@mscherer fedbadges / badges backend on fedora infra is currently in a VM (i.e. not in openshift) AFAICT

should the badges team open a ticket to get a new VM set up for fedbadges like what was done in this ticket for FMN:

https://pagure.io/fedora-infrastructure/issue/9746

from the datanommer side, once fedbadges is updated to python3, fedbages can use the newest version of datanommer.models, and that side of things should just work as it has in the past (the datanommer.models API fedbadges uses hasnt changed)

@ryanlerch
Copy link
Contributor Author

okay, opened a infra ticket for this -- not really sure how to proceed though.

https://pagure.io/fedora-infrastructure/issue/10192

@ryanlerch ryanlerch moved this from Blocked to Done in Datanommer / Datagrepper Sep 6, 2021
@ryanlerch ryanlerch moved this from Done to DOne in Sprint 4 in Datanommer / Datagrepper Sep 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ryanlerch ryanlerch

Labels
L Large Task (about 15 hours)
Projects
Datanommer / Datagrepper
DOne in Sprint 4
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abompard @mscherer @ryanlerch