[Moabusers] problems with showstats -u

[David Backeberg]

We are running moab version 4.2.2b1.

We use
showstats -u

to dump information about user utilization of the cluster, like how
many jobs they've completed and how much time their jobs are getting.
This is also helpful when we help tune job scripts, as we can see when
the users request much more time than what the job requires.

Recently, we brought the Moab service and the machines down for
scheduled maintenance. When the machines came back up, the
showstats -u

output was corrupt, in that some users had now used less time than
before the downtime. Some newer users were no longer listed in the
statistics at all. The output itself is still formatted correctly, but
it's like Moab is somehow ignorant to some of the data it should be
using to calculate these values.

Could somebody please explain more about how Moab actually tracks,
maintains, and updates the information displayed by
showstats -u

We keep our logs in a directory that seems to not have any data
corruptions, but is there a way to prod Moab into describing whether
there are any problems parsing a particular file, or perhaps if some
file may be missing? I found files in /var/spool/moab/stats/ with
names like
DAY.date, events.date, and WEEK.date

I assume Moab stores these stats values daily, and then keeps adding
to the previous day's total, but maybe something more sophisticated is
going on. Please suggest how to troubleshoot my stats problems.

In fact, the DAY.date and WEEK.date files just say <Data></Data> and
don't have any actual data in them other then the pair of tags.

The events.date seem to have the actual useful information, like how
long a job took to complete and who was running it.

Ideas?