Problem with publication information by BDII-Local #135

jas01 · 2015-03-17T16:04:01Z

I got a alert on

midmon.egi.eu

on my bdii about information the bdii publish

CRITICAL - errors 78, warnings 26, info 52
Summary per type of error, warning and info message:
E022 - Default value published (GLUE2ComputingShareWaitingJobs): 26
E023 - Default value published (GLUE2ComputingShareEstimatedAverageWaitingTime): 26
E024 - Default value published (GLUE2ComputingShareEstimatedWorstWaitingTime): 26
I046 - Number of seconds higher than 1 million (GLUE2ComputingShareEstimatedAverageWaitingTime): 26
I047 - Number of seconds higher than 1 million (GLUE2ComputingShareEstimatedWorstWaitingTime): 26
W025 - Incoherent number of total jobs (GLUE2ComputingShareTotalJobs): 26

The text was updated successfully, but these errors were encountered:

Pansanel · 2015-03-18T14:06:29Z

Which is the version of glite-info-provider-service?

jas01 · 2015-03-18T14:12:07Z

Actually on my bdii-local I've

glite-info-provider-service-1.13.4-1.el6.noarch

Regards

Pansanel · 2015-03-18T14:15:50Z

Can you try to downgrade to 1.13.3-1 to see if it works better?

Pansanel · 2015-03-18T14:25:28Z

Dismiss the previous comment. I have found where the code is:
/usr/libexec/info-dynamic-pbs
Which version of lcg-info-dynamic-scheduler-pbs are you using?

jas01 · 2015-03-18T14:30:12Z

The downgrade doen't change anything. But you already known that ;-)

I've split my torque server and the cream-ce.

On both

lcg-info-dynamic-scheduler-pbs-2.4.5-1.el6.noarch

regards.

jrha · 2015-03-18T14:37:39Z

Is this a problem with the Quattor template library or a middleware problem?

Pansanel · 2015-03-18T14:42:18Z

It may be related to Quattor if it is a gip parser configuration problem. It is not sure.
We have the same rpm (lcg-info-dynamic-scheduler-pbs-2.4.5-1.el6.noarch).
Can you check the content of /var/lib/bdii/gip/plugin/lcg-info-dynamic-ce and try to execute it manually?

jas01 · 2015-03-18T14:56:33Z

so in the /var/lib/bdii/gip/plugin/lcg-info-dynamic-ce I've got a very simple

[root@torque-grid ~]# more /var/lib/bdii/gip/plugin/lcg-info-dynamic-ce
#!/bin/sh
cat /swareas/dteam/gip-ce-dynamic-info-maui.cache
[root@torque-grid ~]#

and inside this /swareas/dteam/gip-ce-dynamic-info-maui.cache
I've got lot of information, If I stay on one vo

dn: GlueCEUniqueID=cream-ce-grid.obspm.fr:8443/cream-pbs-glast.org,Mds-Vo-name=resource,o=grid
GlueCEInfoLRMSVersion: 2.5.13
GlueCEInfoTotalCPUs: 112
GlueCEPolicyAssignedJobSlots: 112
GlueCEStateFreeCPUs: 0
GlueCEStateFreeJobSlots: 0
GlueCEPolicyMaxRunningJobs: 110
GlueCEPolicyMaxWallClockTime: 4320
GlueCEPolicyMaxCPUTime: 1440
GlueCEStateStatus: Production

dn: GLUE2ShareID=GLAST-ORG_GLAST-ORG_torque-grid.obspm.fr_ComputingElement,GLUE2ServiceID=torque-grid.obspm.fr_ComputingElement,GLUE2GroupID=resource,o=glue
GLUE2ComputingShareFreeSlots: 0
GLUE2ComputingShareMaxRunningJobs: 110
GLUE2ComputingShareMaxWaitingJobs: 220
GLUE2ComputingShareMaxTotalJobs: 330
GLUE2ComputingShareServingState: Production
GLUE2EntityCreationTime: 2015-03-18T14:55:02Z
GLUE2ComputingShareMaxWallTime: 4320
GLUE2ComputingShareDefaultWallTime: 4320
GLUE2ComputingShareMaxCPUTime: 1440
GLUE2ComputingShareDefaultCPUTime: 1440
GLUE2ComputingShareMaxSlotsPerJob: 1
GLUE2ComputingShareMaxMainMemory: 2000
GLUE2ComputingShareMaxVirtualMemory: 20000

but if I try to find some from the nagios alerte it's doent' exist like

[root@torque-grid ~]# grep GLUE2ComputingShareEstimatedAverageWaitingTime /swareas/dteam/gip-ce-dynamic-info-maui.cache
[root@torque-grid ~]#grep GLUE2ComputingShareEstimatedWorstWaitingTime /swareas/dteam/gip-ce-dynamic-info-maui.cache
[root@torque-grid ~]#

Pansanel · 2015-03-18T18:06:32Z

Which version of template-library-grid are you using? Many work has been done on glue2 static publishing in November. Are you sure you are up to date?

jouvin · 2015-03-18T18:14:35Z

We started with an email exchange with @jas01 and he confirmed me that he was using 14.10.0. As you said, it is not something we see in other sites so probably due to some local conditions but I guess this is coming from some wrong Quattor configuration at site.

We are running the same version of glite-info-provider-service as @jas01 so I'll try to run the GLUE validator on our config again to check that we don't have the same problem...

jouvin · 2015-03-19T11:38:12Z

I have checked and cannot reproduce the problem on our CE... which doesn't mean that there is no problem! I'm trying to think about what could help to troubleshoot the problem...

jouvin · 2015-03-19T11:47:26Z

From what I can see running glue-validator (*), I have the feeling that something is not working with the GIP cache. This cache is made of 2 files: /swareas/dteam/gip-ce-dynamic-info-maui.cache and /swareas/dteam/gip-ce-dynamic-info-scheduler.cache. The problematic information is coming from the latter one (gip-ce-dynamic-info-scheduler.cache). Could you check that this file is created and that you have not error in /var/spool/maui/gip-info-dynamic-scheduler-plugin.sh when running the dynamic-info-scheduler plugin.

(*) glue-validator -H topbdii.grif.fr -p 2170 -b GLUE2DomainID=OBSPM,GLUE2GroupID=grid,o=glue -k -v 3 egi-profile

jas01 · 2015-03-23T14:23:26Z

Sorry I didn't see your email because it's tag as spam.

From what I can see running glue-validator (*), I have the feeling that
something is not working with the GIP cache. This cache is made of 2 files: /
swareas/dteam/gip-ce-dynamic-info-maui.cache and /swareas/dteam/
gip-ce-dynamic-info-scheduler.cache. The problematic information is coming from
the latter one (gip-ce-dynamic-info-scheduler.cache). Could you check that this
file is created and that you have not error in /var/spool/maui/
gip-info-dynamic-scheduler-plugin.sh when running the dynamic-info-scheduler
plugin.

Your right that is the problem. I don't have the symbolic link between

/var/lib/bdii/gip/ldif/static-file-all-CE-pbs.ldif

and

/var/glite/static-file-all-CE-pbs.ldif

I create that symlink manually and every thing seem correct now.

I don't understand how that link disapeare. I try to launch ncm-ncd and he
didn't create this link. Should I modify my own quattor template so it's
create this link ? (for next time I re-install the server)

Regards

pigay · 2015-04-08T15:08:29Z

Dear all,

We came to the same error in midmon (https://midmon.egi.eu/nagios/cgi-bin/extinfo.cgi?host=sitebdii.m3pec.u-bordeaux1.fr&type=2&service=org.bdii.GLUE2-Validate)

We run the same version of lcg-info-dynamic-scheduler-pbs as @jas01 also on 14.10 templates, but we can't find an error running /var/lib/bdii/gip/plugin/lcg-info-dynamic-ce or /var/spool/maui/ gip-info-dynamic-scheduler-plugin.sh.

Can you help us?

pierre

pigay · 2015-04-17T12:36:27Z

Hi,

we investigated a bit further on our problem.

We didn't find any occurence of (for example) GLUE2ComputingShareEstimatedAverageWaitingTimein any of the /usr/libexec/lcg-info-dynamic-* scripts we have on the CE.

we have the following rpms:

lcg-info-dynamic-maui-2.2.0-3.noarch
lcg-info-dynamic-scheduler-pbs-2.4.5-1.el6.noarch

We wonder which piece of software /should/ update these entries with the dynamic value...

Thanks in advance,

Pierre

Pansanel · 2015-04-17T13:06:56Z

Hi Pierre,

Both RPMs are up to date. What is the content of the following file:
/etc/glite-ce-glue2/glite-ce-glue2.conf

Jerome

pigay · 2015-04-17T21:14:24Z

Dear Jérôme,

This file doesn't exist. We only have a /etc/glite-ce-glue2/glite-ce-glue2.conf.template from glite-ce-cream-utils-1.3.5-1.el6.x86_64.rpm

How the .conf file is supposed to be created?

Thanks,

Pierre

jouvin · 2015-04-18T01:27:18Z

Have you checked that the problem is not the same as for @jas01, the missing symlink? This symlink is needed because of a bug introduce in lcg-info-dynamic-scheduler and recently fixed (but may be not yet released).

glite-ce-glue2.conf should exist but in /etc/bdii/gip.

To troubleshoot this kind of problem, you may want to look at /var/log/maui-monitoring.ncm-cron.log. BTW, are you running the CE and the batch system on the same machine? Do you have GIP_CE_USE_CACHE true or false in your configuration? If true, you should have to plugins in /var/lib/bdii/gip/plugin/ on your CE that are just a cat command of some file): check the existence/content of these files.

pigay · 2015-04-26T09:21:31Z

Dear Michel, thanks for helping. Sorry for late reply, I was in holidays.

The symlink looks correct:

[root@ce0 ~]# ls -al /var/lib/bdii/gip/ldif/static-file-all-CE-pbs.ldif /var/glite/static-file-all-CE-pbs.ldif
lrwxrwxrwx 1 root root    50 17 avril 16:49 /var/glite/static-file-all-CE-pbs.ldif -> /var/lib/bdii/gip/ldif/static-file-all-CE-pbs.ldif
-rw-r--r-- 1 ldap ldap 69924 24 déc.  14:46 /var/lib/bdii/gip/ldif/static-file-all-CE-pbs.ldif

/etc/bdii/gip/glite-ce-glue2.conf is present and looks ok, as far as I can see. The file /etc/glite-ce-glue2/glite-ce-glue2.conf mentionned by @Pansanel is missing however.

In /var/log/maui-monitoring.ncm-cron.log, I see that the cron runs every 5 minutes without errors.

We run 2 CEs, each has it's own batch system (and a separate set of WNs). GIP_CE_USE_CACHE is set to true in our templates.

Concerning the plugins, we have:

[root@ce0 ~]# ll /var/lib/bdii/gip/plugin/
total 12
-rwxr-xr-x 1 ldap ldap 49 14 mai    2014 lcg-info-dynamic-ce
-rwxr-xr-x 1 ldap ldap 54 14 mai    2014 lcg-info-dynamic-scheduler-wrapper
-rwxr-xr-x 1 ldap ldap 97 14 mai    2014 lcg-info-dynamic-software-wrapper

Each file contains a cat command (and runs without errors)...

jrha · 2015-04-27T09:22:02Z

I don't mind this kind of discussion here, but is this problem specific to the grid template library?
If not, is there a better forum for this where more people could benefit from the discussion?

pigay · 2015-04-27T14:48:48Z

Dear @jrha, you're probably right. But I think I got it.

@jouvin : by the way, I noticed in quattor-grid mailing list a thread I missed, related to #113. We are not synchronized with this PR (we are in umd3-14.10 I think), so it seems normal that we miss glue2 information...

I'll try to get the correct version and post in quattor-grid if I have trouble.

sorry for the noise.

pigay · 2015-04-28T08:28:13Z

Finally, it turns out to be a template-grid-library issue.

As @jas01 stated before I pollute this thread, this is a missing symlink problem (I had the same issue with /var/glite/ComputingShare.ldif``).

I couldn't find how these links are created in the templates.

jrha · 2015-04-28T10:28:30Z

☺️

jouvin · 2015-05-05T17:45:10Z

I had a look to the problem and proposed fix and I am not completly sure this is the right solution to the problem. I now remember that the problem comes from lcg-info-dynamic-scheduler that in EMI3 releases was ignoring ldif_file and ldif_file_glue2 in its config file (/etc/bdii/gip/lcg-info-dynamic-scheduler-pbs.conf): see https://ggus.eu/index.php?mode=ticket_info&ticket_id=110336 for the explanation.
I would tend to refuse the PR and consider that the correct fix is the new version of lcg-info-dynamic-scheduler. I'll try to assess this before 15.4 release...

…y lcg-info-dynamic-scheduler - See https://ggus.eu/index.php?mode=ticket_info&ticket_id=110336 for details Fixes quattor#135

jouvin · 2015-05-06T17:04:03Z

See #137 for the discussion explaining why it was a workaround but not a real fix for the problem introduced by the need to work around a bug in lcg-info-dynamic-scheduler. #141 should be the real fix.

jouvin · 2015-05-07T14:27:51Z

I have been very bad following this issue, my apologizes... Anyway, just checking the status of everything after comment #141 (comment), I realize that there may be some ambiguity about whether your are running a fixed version of the GIP plugin. The real culprit is not lcg-info-dynamic-sheduler-pbs but the underlying module dynsched-generic. The pb is fixed in 2.5.5-1.
A for the short term, as we are very close to 15.4 release without enough time to test the things properly for something that is difficult to tests (some effects need time to be identified), I suggest that we merge #137 (which is an acceptable workaround) and work quietly on #141 afterwards...
Any objection?

jouvin · 2015-05-07T14:45:50Z

In fact the required version of dynsched-generic has been released only in EMI repos (a while ago)... I try to understand the real status...

jouvin · 2015-05-08T19:36:21Z

For the record, the new version with the bug fix required for this PR will be released by UMD end of may. So let's delay it until 15.6.

jrha · 2015-05-11T09:12:31Z

#137 is a workaround, the real fix should be incorporated into #141.

…y lcg-info-dynamic-scheduler - See https://ggus.eu/index.php?mode=ticket_info&ticket_id=110336 for details Fixes quattor#135

jouvin · 2016-01-18T00:15:46Z

After closing #141 (see comments inside), I propose to restart the work on this issue with the last version of GIP CE as provided in #162 and intended to be released in 16.2. The fixed UMD component should have been available for a long time now and I suggest to remove the workarounds and integrate properly the working version.

jrha · 2016-02-12T12:53:31Z

@jouvin - #162 is merged, what is left to do?

jrha · 2016-02-17T13:04:02Z

@jouvin ping?

jrha · 2016-02-22T10:51:33Z

@jouvin and @jas01 if this is fixed please close this issue, otherwise please move the milestone to 16.4.

jrha added this to the 15.4 milestone Mar 17, 2015

pigay mentioned this issue Apr 29, 2015

fix pbs static ldif paths when GIP_CE_USE_CACHE is true #137

Merged

jouvin mentioned this issue May 6, 2015

GIP CE: fix for problem introduced by workaround for a buggy lcg-info-dynamic-scheduler #141

Closed

jouvin modified the milestones: 15.6, 15.4 May 8, 2015

jrha closed this as completed in #137 May 11, 2015

jrha reopened this May 11, 2015

jrha modified the milestones: 15.10, 15.8 Oct 29, 2015

jrha modified the milestones: 16.2, 15.12 Dec 7, 2015

jouvin self-assigned this Jan 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem with publication information by BDII-Local #135

Problem with publication information by BDII-Local #135

jas01 commented Mar 17, 2015

Pansanel commented Mar 18, 2015

jas01 commented Mar 18, 2015

Pansanel commented Mar 18, 2015

Pansanel commented Mar 18, 2015

jas01 commented Mar 18, 2015

jrha commented Mar 18, 2015

Pansanel commented Mar 18, 2015

jas01 commented Mar 18, 2015

Pansanel commented Mar 18, 2015

jouvin commented Mar 18, 2015

jouvin commented Mar 19, 2015

jouvin commented Mar 19, 2015

jas01 commented Mar 23, 2015

pigay commented Apr 8, 2015

pigay commented Apr 17, 2015

Pansanel commented Apr 17, 2015

pigay commented Apr 17, 2015

jouvin commented Apr 18, 2015

pigay commented Apr 26, 2015

jrha commented Apr 27, 2015

pigay commented Apr 27, 2015

pigay commented Apr 28, 2015

jrha commented Apr 28, 2015

jouvin commented May 5, 2015

jouvin commented May 6, 2015

jouvin commented May 7, 2015

jouvin commented May 7, 2015

jouvin commented May 8, 2015

jrha commented May 11, 2015

jouvin commented Jan 18, 2016

jrha commented Feb 12, 2016

jrha commented Feb 17, 2016

jrha commented Feb 22, 2016

Problem with publication information by BDII-Local #135

Problem with publication information by BDII-Local #135

Comments

jas01 commented Mar 17, 2015

Pansanel commented Mar 18, 2015

jas01 commented Mar 18, 2015

Pansanel commented Mar 18, 2015

Pansanel commented Mar 18, 2015

jas01 commented Mar 18, 2015

jrha commented Mar 18, 2015

Pansanel commented Mar 18, 2015

jas01 commented Mar 18, 2015

Pansanel commented Mar 18, 2015

jouvin commented Mar 18, 2015

jouvin commented Mar 19, 2015

jouvin commented Mar 19, 2015

jas01 commented Mar 23, 2015

pigay commented Apr 8, 2015

pigay commented Apr 17, 2015

Pansanel commented Apr 17, 2015

pigay commented Apr 17, 2015

jouvin commented Apr 18, 2015

pigay commented Apr 26, 2015

jrha commented Apr 27, 2015

pigay commented Apr 27, 2015

pigay commented Apr 28, 2015

jrha commented Apr 28, 2015

jouvin commented May 5, 2015

jouvin commented May 6, 2015

jouvin commented May 7, 2015

jouvin commented May 7, 2015

jouvin commented May 8, 2015

jrha commented May 11, 2015

jouvin commented Jan 18, 2016

jrha commented Feb 12, 2016

jrha commented Feb 17, 2016

jrha commented Feb 22, 2016