Validation Not Working

Message boards : Number crunching : Validation Not Working
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Conan
Avatar

Send message
Joined: 24 Sep 19
Posts: 106
Credit: 827,798
RAC: 405
Australia
Message 589 - Posted: 7 Jun 2020, 9:24:45 UTC

Validation seems to have come to a crawl, over 145,000 work units awaiting validation at the moment.

Conan
ID: 589 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 591 - Posted: 8 Jun 2020, 4:59:46 UTC

Yes.
We have temporary performance problems with database optimization/cleaning...
ID: 591 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Conan
Avatar

Send message
Joined: 24 Sep 19
Posts: 106
Credit: 827,798
RAC: 405
Australia
Message 644 - Posted: 1 Jul 2020, 23:27:58 UTC
Last modified: 1 Jul 2020, 23:28:55 UTC

Tasks ready to send 170142
Tasks in progress 3886
Workunits waiting for validation 981494


981,000+ work units waiting for validation, seems to be a problem somewhere.

Conan
ID: 644 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Conan
Avatar

Send message
Joined: 24 Sep 19
Posts: 106
Credit: 827,798
RAC: 405
Australia
Message 645 - Posted: 2 Jul 2020, 3:10:14 UTC - in response to Message 644.  

Tasks ready to send 170142
Tasks in progress 3886
Workunits waiting for validation 981494


981,000+ work units waiting for validation, seems to be a problem somewhere.

Conan


We have topped 1,000,000 WUs waiting for validation

Computing status
Work
Tasks ready to send 183457
Tasks in progress 4120
Workunits waiting for validation 1028104

Conan
ID: 645 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Volunteer moderator

Send message
Joined: 7 Apr 20
Posts: 430
Credit: 1,119,418
RAC: 439
United States
Message 646 - Posted: 2 Jul 2020, 16:40:40 UTC

Validation is slowly starting to go down a bit:

Tasks ready to send 461715
Tasks in progress 1862
Workunits waiting for validation 971878
Workunits waiting for assimilation 45
Workunits waiting for file deletion 16
Tasks waiting for file deletion 3
Transitioner backlog (hours) 13.91

Task data as of 2 Jul 2020, 15:47:05 UTC

NOTE : The project status page is almost 1 hour behind the real time

Also several of my PC's are NOT getting work despite requesting it over and over.... Just one example:
02-Jul-2020 11:32:32 [iThena] Sending scheduler request: To fetch work.
02-Jul-2020 11:32:32 [iThena] Requesting new tasks for CPU
02-Jul-2020 11:32:36 [iThena] Scheduler request completed: got 0 new tasks
02-Jul-2020 11:32:36 [iThena] No tasks sent


ID: 646 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 648 - Posted: 2 Jul 2020, 20:10:49 UTC

Validator is running...
But this is a performance problem.
I'm trying to improve performance in this weekend...
ID: 648 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Conan
Avatar

Send message
Joined: 24 Sep 19
Posts: 106
Credit: 827,798
RAC: 405
Australia
Message 649 - Posted: 2 Jul 2020, 22:32:54 UTC

Thanks Rysiu, always a new issue to keep things interesting, eh?

Conan
ID: 649 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ChelseaOilman

Send message
Joined: 26 Dec 19
Posts: 2
Credit: 25,037,694
RAC: 33,816
United States
Message 651 - Posted: 3 Jul 2020, 15:11:55 UTC

I have twice as many tasks waiting to be validated than tasks that have been validated.

Validation pending (2379) · Validation inconclusive (0) · Valid (1156) · Invalid (0) · Error (0)
ID: 651 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mikey

Send message
Joined: 12 May 20
Posts: 29
Credit: 1,681,938
RAC: 1,303
United States
Message 652 - Posted: 3 Jul 2020, 21:13:27 UTC - in response to Message 651.  

I have twice as many tasks waiting to be validated than tasks that have been validated.

Validation pending (2379) · Validation inconclusive (0) · Valid (1156) · Invalid (0) · Error (0)


Mine are different:
All (5087) · In progress (10) · Validation pending (2756) · Validation inconclusive (0) · Valid (2319) Invalid (0) · Error (2)
ID: 652 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 654 - Posted: 4 Jul 2020, 12:10:01 UTC

I will probably have to stop generating new ina PERF tasks for some time.
For some time these tasks may be unavailable.
I will try to reduce these problems.

I'm still analyzing the situation.
However, I will probably have to reduce the number of tasks temporarily.
Then the situation with the availability of the tasks will return to normal.

Also, in general, there will probably soon be an update of Thena PERF.
ID: 654 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 668 - Posted: 6 Jul 2020, 13:08:44 UTC

The number of tasks in the validation queue has started to decrease.
I added four parallel validators (mod 4).
Current status:
Workunits waiting for validation 1679395
ID: 668 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Sergey Kovalchuk

Send message
Joined: 23 Sep 19
Posts: 59
Credit: 789,767
RAC: 0
Ukraine
Message 669 - Posted: 6 Jul 2020, 15:41:57 UTC - in response to Message 668.  

https://boinc.berkeley.edu/trac/wiki/BackendPrograms

The transitioner works asynchronously from work generators and schedulers, and in high-load situations it can fall behind (i.e. a large queue of workunits waiting to be processed can build up). The transitioner backlog is shown in the server status page. If the backlog becomes large (> 1 hour) you can do the following:

    •stop the project
    •run sched/transitioner_catchup.php
    •start the project

ID: 669 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Conan
Avatar

Send message
Joined: 24 Sep 19
Posts: 106
Credit: 827,798
RAC: 405
Australia
Message 673 - Posted: 9 Jul 2020, 10:10:43 UTC
Last modified: 9 Jul 2020, 10:17:39 UTC

Just curious about the numbers shown on the server for validation.

It shows at the moment just 133 awaiting validation across the whole project.

I have 317 currently awaiting validation on my account page and it is going up.

Also the last validation that happened on my computers was on the 7th of July, so a bit behind.

Just wondering how it works.

Conan
ID: 673 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : Validation Not Working

© 2019-2024 iThena. All rights reserved. | Private Policy

Page generated on 29 Mar 2024, 10:45:19 UTC in 0.1903 seconds.