Домой United States dc3 United States software in english pdf Reshaping PyTorch Tensors

Reshaping PyTorch Tensors

368
0
ПОДЕЛИТЬСЯ

PyTorch is an extensive tool for deep learning. Let’s discuss how the central data structure the Tensor is designed. What are the pitfalls of a simple operation
Join the DZone community and get the full member experience. It is a reasonable thing to expect n-dimensional tensor to have a possibility to be reshaped. Reshape means to change the spatial size of a container that holds underlying data. One can create any n-dimensional tensor that wraps a numerical array as long as the product of dimensions stays equal to the number of the array’s elements. The same holds for reshaping. It’s possible to convert the existing container to a different spatial size while it preserves the total number of elements. This is quite intuitive. But, the experienced users might notice a couple of not obvious things. The answer to the first question is that reshaped tensor sometimes triggers copying of underlying data and sometimes doesn’t. The documentation says the following about reshape method: This quote introduces three things: the view of a tensor, the stride of a tensor, and contiguous tensors. Let’s talk about them in order. Stride is a tuple of integers each of which represents a jump needed to perform on the underlying data to go to the next element within the current dimension. The tensor’s underlying data is just a one-dimensional physical array, that is stored sequentially in the memory. If we have, for example, an arbitrary 3-dimensional container, then the elements at indices (x,y,z) and at (x+1,y,z) live at a distance of size stride(0) within the underlying data. See the code below: Using all the stride entries we can translate any given n-dimensional index to the offset within a physical array by ourselves. Here’s the graphical interpretation of a stride. This image is cherry-picked from ezyang’s blog, one of the core PyTorch maintainers. In the picture above, you can see how the index (1,0) inside a 2-by-2 logical tensor translates to the offset of a value 2 within a physical array using strides. First, consider a one-dimensional array. One might say it is contiguous if the entries are stored sequentially without any spaces in between. Visualizations from this StackOverflow answer will help us: Then, consider a multi-dimensional array. A special case is two dimensions,2d array is contiguous if the underlying data is stored row-wise sequentially. We can generalize that by saying the n-dimensional array is contiguous if its (n-1)-dimensional components are contiguous and stored sequentially in memory. Finally, we can say that a PyTorch tensor is contiguous if the n-dimensional array it represents is contiguous. One can deduce the contiguity from the tensor strides. If we take an arbitrary n-dimensional tensor that is contiguous then each stride in the tuple is the product of the corresponding tail of tensor sizes. As in the example: Another interesting aspect is that some of the view operators return a new tensor object based on the same underlying data but treat it in a non-contiguous way. For example, transposition. But there’s a way of restoring contiguity using the method contiguous on a tensor, which might trigger copying if the tensor was non-contiguous. Contiguity is crucial when reshaping is performed. If the tensor is not contiguous then reshape will silently trigger data copying which is not efficient but in some cases it’s unavoidable. The code above shows the cases when the copying happens silently because of non-contiguous order. This method is aimed to reshape tensor efficiently, when possible. That means to use underlying data without explicit copying. The reshape itself is performed by updating the shape and stride of a tensor. The rule of thumb is the following the tensor could be reshaped by view if it is_contiguous==True. Let’s have a look at this example: Here are two tests. The one on contiguous tensor and the other on non-contiguous, it is seen that in the latter case view is impossible and it recommends us to use reshape instead. Here it is! That much theory we need to correctly interpret the documentation on reshape function. And we are ready to answer the first question stated at the beginning: does reshape explicitly copy underlying data? Reshape method applied on a tensor will try to invoke the view method if it is possible, if not then the tensor data will be copied to be contiguous, i.e. to live in the memory sequentially and to have proper strides, and after this manipulation, it will invoke the view. It is might be clear now that indexing within a reshaped tensor hardly relies on the stride. When we perform reshaping or view the tensor shape and stride properties change. This two are sufficient to deduce where the elements do live after the modification. In this short article, I tried to accumulate the knowledge on tensor reshaping in PyTorch, the thing that sometimes arise nagging questions during my day-to-day work. It is obvious that copying of underlying memory is bad and it is better to have reshaping without it, but sometimes one might come across it when it happens silently (for example on transposed tensors). Of course, you can learn the use-cases when there’s a place to be, but from my perspective, it is better to understand hidden design, which is quite elegant. In the next article, I’d like to talk about broadcasting with tensors: how it works and when it is not efficient. [1] Thumbnail photo source Opinions expressed by DZone contributors are their own.