NMF - distributed control doesn't land as I throw more nodes

User 316 | 7/17/2014, 2:33:58 AM

I've noticed that as I throw more nodes (> 8 nodes | 16 cores each) at the NMF algorithm the distributed control doesn't land and the application hangs after saving the last V matrix output file. Does anyone had the same kind of issue?

ESC[0mINFO: distributedingressbase.hpp(finalize:244): Skipping Graph Finalization because no changes happened... ESC[0mINFO: distributedingressbase.hpp(finalize:244): Skipping Graph Finalization because no changes happened... ESC[0mINFO: distributedingressbase.hpp(finalize:244): Skipping Graph Finalization because no changes happened... ESC[0mINFO: distributedingressbase.hpp(finalize:244): Skipping Graph Finalization because no changes happened... ESC[0mINFO: distributedingressbase.hpp(finalize:244): Skipping Graph Finalization because no changes happened... ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.1of_8 ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.8of_8 ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.6of_8 ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.7of_8 ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.2of_8 ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.3of_8 ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.5of_8 ESC[0mINFO: distributedgraph.hpp(savetoposixfs:1856): Saving to file: /output.V.4of_8 ESC[0mAPPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)

No Comments