Week2 Update:
Apologies for the delayed post of update of week 1 and week 2 progress. It is two weeks into the coding phase of GSoC and my experience has been greatly overwhelming. I have been learning a lot of new things everyday, and I am facing challenging tasks that keeps my motivation high to work learn more.
In the first two weeks, I have built a new Global Optimizer in theano. This new global optimizer builds a graph in parallel to the existing graph and makes the transfer of the OPs to the GPU in one single pass. The new optimizer has been performing pretty well as of now and currently I am analysing the time this optimizer takes per optimizing each nodes and working on speeding up the slower ones.
This optimizer is giving some amazing results. It halves the time taken for optimization. I am currently profiling the optimizer on Sparse Bayesian Recurrent Neural Network model. The optimization takes 32.56 seconds to reduce 10599 to 9404 nodes with the new optimizer, whereas it used to take 67.7 seconds for reducing 10599 nodes to 9143 nodes with the old optimizer. This has been a pleasing result.
Also, along the way, there were few more tasks done to speed up the existing optimizers. One such is, reducing the number of instance a new instance of an Op is created by pre-instantiating them. This helped in speeding up by ~2sec. Another speedup task was caching the GPUAlloc class(the class that is used to create initialized memory on the GPU). This has reduced the optimizer timing by ~3sec.
I had a few roadblocks while building this new optimizer. Thanks to my mentor (Frèdèric) who helped me out to get out few of them. It has been an amazing time to work with a highly knowledgable, and an experienced person.
I am profiling the results this new optimizer is giving on few Deep Learning models to evaluate its overall performance. In the next few days, I will write an other blog post elaborating on the profiling result of this optimizer and make this optimizer work with models that take highly unusual time to compile time with the current optimizers if the model parameters are humongous.
That's it for now folks, stay tuned!
Apologies for the delayed post of update of week 1 and week 2 progress. It is two weeks into the coding phase of GSoC and my experience has been greatly overwhelming. I have been learning a lot of new things everyday, and I am facing challenging tasks that keeps my motivation high to work learn more.
In the first two weeks, I have built a new Global Optimizer in theano. This new global optimizer builds a graph in parallel to the existing graph and makes the transfer of the OPs to the GPU in one single pass. The new optimizer has been performing pretty well as of now and currently I am analysing the time this optimizer takes per optimizing each nodes and working on speeding up the slower ones.
This optimizer is giving some amazing results. It halves the time taken for optimization. I am currently profiling the optimizer on Sparse Bayesian Recurrent Neural Network model. The optimization takes 32.56 seconds to reduce 10599 to 9404 nodes with the new optimizer, whereas it used to take 67.7 seconds for reducing 10599 nodes to 9143 nodes with the old optimizer. This has been a pleasing result.
Also, along the way, there were few more tasks done to speed up the existing optimizers. One such is, reducing the number of instance a new instance of an Op is created by pre-instantiating them. This helped in speeding up by ~2sec. Another speedup task was caching the GPUAlloc class(the class that is used to create initialized memory on the GPU). This has reduced the optimizer timing by ~3sec.
I had a few roadblocks while building this new optimizer. Thanks to my mentor (Frèdèric) who helped me out to get out few of them. It has been an amazing time to work with a highly knowledgable, and an experienced person.
I am profiling the results this new optimizer is giving on few Deep Learning models to evaluate its overall performance. In the next few days, I will write an other blog post elaborating on the profiling result of this optimizer and make this optimizer work with models that take highly unusual time to compile time with the current optimizers if the model parameters are humongous.
That's it for now folks, stay tuned!
No comments:
Post a Comment