Predictive services

User 2192 | 8/27/2015, 8:23:00 AM

Hi,

I've tried to deploy it onto an ec2 instance, following the instaruction, but I get an error that my '/tmp' folder is full and the model is not successfully deployed. So, when calling: deployment.get_status() I get : [{'cache': {u'healthy': True, u'numkeys': 0, u'type': u'local'}, 'dnsname': u'ec2-54-177-42-200.us-west-1.compute.amazonaws.com', 'id': u'i-bff40a7a', 'models': [{u'cacheenabled': False, u'description': u'', u'name': u'recs', u'reason': u"[Errno 28] No space left on device: '/tmp/predictiveobject_xpKEFi'", u'status': u'Failed to load', u'version': 0}], 'reason': u'N/A', 'state': u'InService'}]

Is there a way to route the folder to another location where there is more free space ? Also, is there any way I can delete the contents of this folder ? Currently, I can't even access this instance since it has no key pair defined.

Thanks, Andrea.

Comments

User 1178 | 8/30/2015, 5:43:53 PM

Hi alkandrea,

For now, there is no easy way for you to log into the service to make modifications to it. We also do not encourage customers to do that. The /tmp space full is most likely caused by big models you have. How big is your model?

The suggestion I have for now is to use Predictive Service's replace_nodes() API to replace the node that has /tmp space full. With this, the new node will be started with a clean slate.

You may try try to check out the system state by using a custom function. Here is an example: You may replace cmd content with something you would like to see. But be careful to make any modifications to the system here.

   def debugger(cmd=None):
	  import subprocess
	  if cmd:
	    return subprocess.check_output(cmd, shell=True)
	  else:
	    return 'did nothing'
   ps.add('debugger', debugger)
   ps.apply_changes()
   result =  ps.query('debugger', cmd = 'du -h /tmp')
   print result['response']

We are working on to make sure files that are no longer used get cleaned up automatically.

Let me know how it goes.

Thanks!

Ping


User 2192 | 8/31/2015, 8:09:33 AM

Hi Ping,

Thanks for the replay :)

I have tried to replace the node, but on the newly create node I get the following status: 'state': u'OutOfService'

[ERROR] Cannot get node status from, error: Cannot get status for host, error: HTTPConnectionPool(host='', port=9005): Read timed out. (read timeout=10)

What can cause this?

Thanks, Andrea.


User 2192 | 8/31/2015, 12:56:22 PM

Hi,

I was able to replace the instance eventually but I still get into the same issue where the disk is full.

Any way to overcome this ?

Thanks, Andrea.


User 1178 | 8/31/2015, 9:20:49 PM

Hi Andrea,

I will follow up with you through email to get more information regarding your setup.

Thanks

Ping