Add a progress bar to Job.get_status() if it is blocked waiting

User 2568 | 3/16/2016, 10:12:10 AM

The hyper parameter search stuff is helpful, but I'd really like some kind of feedback. Ideal a progress bar and perhaps estimate of time to complete. Here is a simple example.

from IPython.html.widgets import FloatProgress
from IPython.display import display
from time import sleep

total = job.get_status()['Completed'] + job.get_status()['Running'] + job.get_status()['Pending']

f = FloatProgress(min=0, max=total)
display(f)
while running > 0:
    f.value = total-job.get_status()['Running'] + job.get_status()['Pending']
    time.sleep(60)

the only problem with this all the jobs run at the same time, so no progress is seen.

Comments

User 19 | 3/16/2016, 5:40:47 PM

Hi Kevin,

We agree! We will add this as a feature request.

Chris


User 2568 | 3/17/2016, 2:22:40 AM

Here is a poor mans version of a job status bar that works surprisingly well.

from ipywidgets import FloatProgress
from IPython.display import display
from time import sleep
from os import listdir

def complete_job(job):
    '''Calculate an accurate, fine grained count of complete jobs'''

    #note Job.get_status() is very coarse grained and based on whether the entire job is done or not.

    complete = 0
    for i in range(0,len(job.jobs)):
        #This assumes either one of the output files is an running task, or its the _combine_mps_tasks
        complete += len(listdir(job.jobs[i]._exec_dir + "/output"))-1

    return complete

def job_status_bar(job):
    '''Display a status bar showing what % of the work has been done'''

    #Calculate teh total number of tasks being run across the jobs.
    total = job.get_status()['Completed'] + job.get_status()['Running'] + job.get_status()['Pending']

    f = FloatProgress(min=0, max=total, description="Completed Jobs",bar_style='success')
    display(f)
    complete=0
    while complete < total:
        complete = complete_job(job)
        f.value = complete
        sleep(1)

User 2568 | 3/17/2016, 2:41:52 AM

Here is an improved version

from ipywidgets import FloatProgress
from IPython.display import display
from time import sleep
from os import listdir

def complete_job(job):
    '''Calculate and accurate, fine grained count of complete jobs'''

    #note Job.get_status() is inaccurate and based on wheather the entier job is done or not.   
    return sum([len(listdir(j._exec_dir + "/output"))-1 for j in job.jobs])


def job_status_bar(job):
    '''Display a status bar showing what % of the work has been done'''

    total = sum([len(j._stages[0]) for j in job.jobs])

    f = FloatProgress(min=0, max=total, description="Completed Jobs",bar_style='success')
    display(f)
    complete=0
    while complete < total:
        complete = complete_job(job)
        f.value = complete
        sleep(1)

User 2568 | 3/17/2016, 2:45:26 AM

Of course, what would be really nice is a prediction of the time to completion based on the the time to compute the completed tasks so far.


User 2568 | 3/17/2016, 4:53:32 AM

This now update total number of tasks left.

def completed_jobs(job):
    '''Calculate and accurate, fine grained count of complete jobs'''

    #note Job.get_status() is inaccurate and based on wheather the entier job is done or not.   
    return sum([len(listdir(j._exec_dir + "/output"))-1 for j in job.jobs])


def job_status_bar(job):
    '''Display a status bar showing what % of the work has been done'''

    total = sum([len(j._stages[0]) for j in job.jobs])

    f = FloatProgress(min=0, max=total, description="{:1.0f} tasks left ".format(total), bar_style='success')
    f.value=completed_jobs(job)
    display(f)

    while f.value < total:
        f.value = completed_jobs(job)
        f.description =  "{:1.0f} tasks left ".format(total - f.value)
        sleep(1)

User 1190 | 3/17/2016, 8:04:34 PM

Kevin,

Thank you so much for sharing your code for this much needed feature. Since the modelparametersearch code is built in python, you are welcome to hack the source and make your own modelparametersearch experience.

We would like to incorporate it into the modelparametersearch soon.

Best, -jay


User 2568 | 3/18/2016, 4:04:08 AM

I've done this as noted in this discussion