Server is out of disk space

Message boards : Number crunching : Server is out of disk space
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 24 · Next

AuthorMessage
morgan

Send message
Joined: 30 Sep 19
Posts: 8
Credit: 562,096
RAC: 116
Norway
Message 626 - Posted: 27 Jun 2020, 11:19:02 UTC

Upload is failing, getting ; Server is out of disk space.
ID: 626 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Steve Dodd

Send message
Joined: 4 Apr 20
Posts: 25
Credit: 607,418
RAC: 501
United States
Message 628 - Posted: 27 Jun 2020, 22:45:43 UTC

Someone needs to kick the db_purge processes. :)
ID: 628 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 631 - Posted: 28 Jun 2020, 17:04:36 UTC

I make changes and fix errors...
ID: 631 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 634 - Posted: 28 Jun 2020, 18:35:31 UTC

I'm cleaning the space on one of the partitions...
The process may take some time...
I'm monitoring the problem.
ID: 634 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 635 - Posted: 28 Jun 2020, 21:52:50 UTC

Storage problem solved.
ID: 635 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mikey

Send message
Joined: 12 May 20
Posts: 29
Credit: 1,681,408
RAC: 1,385
United States
Message 638 - Posted: 29 Jun 2020, 2:22:26 UTC - in response to Message 635.  

Storage problem solved.


Thank you very much!!!
ID: 638 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kotenok2000

Send message
Joined: 22 May 20
Posts: 10
Credit: 218,244
RAC: 159
Russia
Message 670 - Posted: 8 Jul 2020, 0:29:05 UTC

It happened again
ID: 670 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Volunteer moderator

Send message
Joined: 7 Apr 20
Posts: 429
Credit: 1,119,176
RAC: 461
United States
Message 671 - Posted: 8 Jul 2020, 0:37:23 UTC - in response to Message 638.  
Last modified: 8 Jul 2020, 0:40:22 UTC

Yep Full
7/7/2020 19:33:48 | iThena | [error] Error reported by file upload server: can't open file PERF_TESTS_0C_5090894_2_30_0_r1041391138_0: No space left on device

Server Status as of 7 Jul 2020, 23:28:35 UTC

Tasks ready to send	107148
Tasks in progress	4604
Workunits waiting for validation	367189
Workunits waiting for assimilation	647718
Workunits waiting for file deletion	11
Tasks waiting for file deletion	5
Transitioner backlog (hours)	21.83

sample_assimilator (ithena_cnode)	boinc-server	Not Running
sample_assimilator_perf (ithena_perf)	boinc-server	Not Running


ID: 671 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 672 - Posted: 8 Jul 2020, 16:59:37 UTC

Yes.
The problem has come up again.
I'm trying to control the situation.
ID: 672 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Conan
Avatar

Send message
Joined: 24 Sep 19
Posts: 106
Credit: 827,542
RAC: 422
Australia
Message 674 - Posted: 9 Jul 2020, 10:19:32 UTC

Might be why uploads are stalling for some time before finally uploading and the web pages are slow to load.

Conan
ID: 674 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Volunteer moderator

Send message
Joined: 7 Apr 20
Posts: 429
Credit: 1,119,176
RAC: 461
United States
Message 677 - Posted: 10 Jul 2020, 18:02:22 UTC

Still seeing the server full / can't upload error messages.
7/10/2020 13:02:15 | iThena | [error] Error reported by file upload server: can't open file PERF_TESTS_0C_6370752_2_30_0_r890170402_0: No space left on device

Server still appears to be overworked/overloaded:
Task data as of 10 Jul 2020, 16:44:02 UTC 
Tasks ready to send		329294
Tasks in progress	5381
Workunits waiting for validation	90763
Workunits waiting for assimilation	1775037
Workunits waiting for file deletion	2
Tasks waiting for file deletion	16
Transitioner backlog (hours)	26.80


ID: 677 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Rysiu
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 25 Aug 19
Posts: 409
Credit: 1,240,185
RAC: 0
Poland
Message 678 - Posted: 10 Jul 2020, 19:51:46 UTC

Now the problem is on the assimilator deamon.

The problem with the disk space is a little different.
It's not typically disk space.

I made some design/implementation bugs in the design.
The problem is the number of files on the partition.
I'm currently removing this problem.

There may still be 'No space left on device' errors.
I think this problem will be gone by this Sunday.
ID: 678 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Volunteer moderator

Send message
Joined: 7 Apr 20
Posts: 429
Credit: 1,119,176
RAC: 461
United States
Message 679 - Posted: 10 Jul 2020, 19:58:31 UTC - in response to Message 678.  

Rysiu,

Thanks for the quick update!
Keep up the good work and hope you can get the gremlins out of the project.

ID: 679 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mikey

Send message
Joined: 12 May 20
Posts: 29
Credit: 1,681,408
RAC: 1,385
United States
Message 680 - Posted: 10 Jul 2020, 23:35:06 UTC - in response to Message 678.  

Now the problem is on the assimilator deamon.

The problem with the disk space is a little different.
It's not typically disk space.

I made some design/implementation bugs in the design.
The problem is the number of files on the partition.
I'm currently removing this problem.

There may still be 'No space left on device' errors.
I think this problem will be gone by this Sunday.


THANK YOU!!!
ID: 680 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Conan
Avatar

Send message
Joined: 24 Sep 19
Posts: 106
Credit: 827,542
RAC: 422
Australia
Message 724 - Posted: 25 Jul 2020, 3:25:30 UTC

G'Day Rysiu,

Just letting you know that this issue has raised it's ugly head again.

Can't upload any work.

Conan
ID: 724 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
UBT - Timbo

Send message
Joined: 10 May 20
Posts: 4
Credit: 30,302
RAC: 0
United Kingdom
Message 725 - Posted: 25 Jul 2020, 7:52:06 UTC - in response to Message 724.  

Same issue here:

25/07/2020 08:51:05 | iThena | [error] Error reported by file upload server: can't write file PERF_TESTS_0D_9653604_2_30_0_r1586974986_0: No space left on server
ID: 725 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
M0CZY

Send message
Joined: 13 Nov 19
Posts: 17
Credit: 187,576
RAC: 70
United Kingdom
Message 800 - Posted: 11 Sep 2020, 14:52:17 UTC - in response to Message 725.  

Fri 11 Sep 2020 15:50:05 BST | iThena | [error] Error reported by file upload server: Server is out of disk space
ID: 800 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Volunteer moderator

Send message
Joined: 7 Apr 20
Posts: 429
Credit: 1,119,176
RAC: 461
United States
Message 801 - Posted: 11 Sep 2020, 17:33:37 UTC
Last modified: 11 Sep 2020, 17:34:07 UTC

Looks the the SEVER IS OUT OF SPACE AGAIN!
Add 5 hours to times below to get UTC
9/11/2020 12:07:37 | iThena | [error] Error reported by file upload server: can't write file PERF_TESTS_0D_5705906_2_30_0_r1975168869_0: No space left on server
9/11/2020 12:07:37 | iThena | Temporarily failed upload of PERF_TESTS_0D_5705906_2_30_0_r1975168869_0: transient upload error
9/11/2020 12:07:37 | iThena | Backing off 03:09:33 on upload of PERF_TESTS_0D_5705906_2_30_0_r1975168869_0


ID: 801 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Volunteer moderator

Send message
Joined: 7 Apr 20
Posts: 429
Credit: 1,119,176
RAC: 461
United States
Message 802 - Posted: 11 Sep 2020, 21:20:24 UTC

Not getting better.
I currently have 426 tasks waiting for validation and growing, still getting the "Error reported by file upload server: can't write file".

SERVER STATS seem wonky:
Computing status
Task data as of 11 Sep 2020, 20:23:48 UTC
Tasks ready to send	204312
Tasks in progress	7320
Workunits waiting for validation	129
Workunits waiting for assimilation	27
Workunits waiting for file deletion	49
Tasks waiting for file deletion	52
Transitioner backlog (hours)	26.86


ID: 802 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Volunteer moderator

Send message
Joined: 7 Apr 20
Posts: 429
Credit: 1,119,176
RAC: 461
United States
Message 804 - Posted: 12 Sep 2020, 18:16:35 UTC

My failed uploads finally went through but my Validation pending shot up to 353 tasks.

Server Status page still showing HIGH hours of backlog and db_purge2 (ithena_perf) - boinc-server is Not Running:
Task data as of 12 Sep 2020, 17:47:50 UTC
Tasks ready to send	837294
Tasks in progress	4996
Workunits waiting for validation	133
Workunits waiting for assimilation	39
Workunits waiting for file deletion	0
Tasks waiting for file deletion	11
Transitioner backlog (hours)	39.53


Hope the welcome and mention of this as a NEW project on the BOINC BOARDS will not crash the system when the influx of new users happens.

ID: 804 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · 3 · 4 . . . 24 · Next

Message boards : Number crunching : Server is out of disk space

© 2019-2024 iThena. All rights reserved. | Private Policy

Page generated on 28 Mar 2024, 21:16:17 UTC in 0.2428 seconds.