Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture.pdf

章芋文

14页

0次

2022-08-29

免费下载

Large Graph Convolutional Network Training with

GPU-Oriented Data Communication Architecture

Seung Won Min

UIUC

min16@illinois.edu

Kun Wu

UIUC

kunwu2@illinois.edu

Sitao Huang

UIUC

shuang91@illinois.edu

Mert Hidayetoğlu

UIUC

hidayet2@illinois.edu

Jinjun Xiong

IBM T.J. Watson Research Center

jinjun@us.ibm.com

Eiman Ebrahimi

NVIDIA

eebrahimi@nvidia.com

Deming Chen

UIUC

dchen@illinois.edu

Wen-mei Hwu

NVIDIA / UIUC

whwu@nvidia.com

ABSTRACT

Graph Convolutional Networks (GCNs) are increasingly adopted

in large-scale graph-based recommender systems. Training GCN

requires the minibatch generator traversing graphs and sampling

the sparsely located neighboring nodes to obtain their features.

Since real-world graphs often exceed the capacity of GPU memory,

current GCN training systems keep the feature table in host memory

and rely on the CPU to collect sparse features before sending them

to the GPUs. This approach, however, puts tremendous pressure

on host memory bandwidth and the CPU. This is because the CPU

needs to (1) read sparse features from memory, (2) write features

into memory as a dense format, and (3) transfer the features from

memory to the GPUs.

In this work, we propose a novel GPU-oriented data commu-

nication approach for GCN training, where GPU threads directly

access sparse features in host memory through zero-copy accesses

without much CPU help. By removing the CPU gathering stage,

our method signicantly reduces the consumption of the host re-

sources and data access latency. We further present two important

techniques to achieve high host memory access eciency by the

GPU: (1) automatic data access address alignment to maximize PCIe

packet eciency, and (2) asynchronous zero-copy access and kernel

execution to fully overlap data transfer with training. We incorpo-

rate our method into PyTorch and evaluate its eectiveness using

several graphs with sizes up to 111 million nodes and 1.6 billion

edges. In a multi-GPU training setup, our method is 65-92% faster

than the conventional data transfer method, and can even match

the performance of all-in-GPU-memory training for some graphs

that t in GPU memory.

PVLDB Reference Format:

Seung Won Min, Kun Wu, Sitao Huang, Mert Hidayetoğlu, Jinjun Xiong,

Eiman Ebrahimi, Deming Chen, and Wen-mei Hwu. Large Graph

Convolutional Network Training with GPU-Oriented Data Communication

Architecture. PVLDB, 14(11): 2087 - 2100, 2021.

doi:10.14778/3476249.3476264

This work is licensed under the Creative Commons BY-NC-ND 4.0 International

License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of

this license. For any use beyond those covered by this license, obtain permission by

emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights

licensed to the VLDB Endowment.

Proceedings of the VLDB Endowment, Vol. 14, No. 11 ISSN 2150-8097.

Host Memory

Block

Transfer

(a) Dense Feature (e.g. Image)

(b) Sparse Feature (e.g. Graph)

GPU

Host Memory

GPU

PCIe PCIe

Figure 1: Challenge of GPUs accessing ne-grained sparse

features in host memory.

PVLDB Artifact Availability:

The source code, data, and/or other artifacts have been made available at

https://github.com/K-Wu/pytorch-direct_dgl.

1 INTRODUCTION

Acceleration of modern machine learning models is often severely

limited by insucient memory bandwidth [

]. To provide

the best possible memory bandwidth, data is usually placed in

memory closest to the processing units of the accelerators [

54]. However, with extremely large datasets, it is inevitable to put

data farther from the processing units to take advantage of larger

capacity (e.g., host memory). In this case directly accessing remote

data from the processing units can be very inecient due to slow

external interconnects. To free processing units from spending

excessive amount of time accessing remote data, modern hardware

systems utilize direct memory access (DMA) engines.

DMA engines are specialized in transferring large blocks of

data independently. By providing source and destination memory

pointers along with the data size, DMA engines transfer data behind

the scenes while keeping processing units available for other tasks.

Initiating each DMA requires multiple interactions between the

user application and the operating system, but these overheads can

be oset by transferring large data blocks (Figure 1 (a)).

The recent adaptation of machine learning to a wide range of

tasks has led modern deep neural networks to work on more com-

plicated data structures such as graphs. Graphs are essential in

doi:10.14778/3476249.3476264

2087

Node 2

Node 4

Node 7

Node 9

Node 25

(a) Neighbor Aggregation (b) Node Feature Tensor

Node Feature

Figure 2: (a) A simple example of GCN training on single

node. (b) An illustration of node features in memory. The

neighboring nodes’ features are scattered in memory.

representing real-world relational information in social networks

and e-commerce. The capability to build high-quality recommender

systems on graphs is indispensable to multiple businesses. In these

graph data structures, the data which we need to access is often

not coalesced together, but scattered in memory (Figure 1 (b)).

One of the most successful adaptations of deep neural network

models to graph data is Graph Convolutional Network (GCN) [

The core idea of GCN is to create node embeddings by iteratively

aggregating neighboring nodes’ attributes using neural networks.

Due to its neighboring node’s attribute lookup, training GCN re-

quires accessing multiple scattered locations in memory. In Figure 2

(a), we show a simple example of GCN training. To generate the

embedding of node 4, we traverse the input graph and aggregate

node 4’s features alongside the features of all neighboring nodes in

the node feature tensor. The example that we show here is only a

toy example. In real-world graphs, each node can be connected to

thousands of nodes. To collect relational information from those

neighboring nodes, we may need to access thousands of scattered

locations in memory. Without a doubt, such data access patterns

make the traditional block data transfer method ineective.

In this work, we propose a processor-oriented, software-dened

data communication architecture. Instead of using DMA engines,

we program GPU cores to directly access host memory with zero-

copy memory access. This approach allows the application devel-

opers to direct the GPU cores to exactly the locations that hold

the data needed for computation. Conventional wisdom may still

argue that since the node feature data is in host memory, CPU has

signicant bandwidth advantage over GPUs and therefore DMA

should be a better option because CPU can quickly gather the sparse

features on the y. However, recent work has shown that the ability

to issue a massive number of concurrent memory accesses enables

GPUs to tolerate latency eectively when accessing complicated

data structures like graphs that reside in host memory [

]. There-

fore, in GCN training, if GPUs can make targeted ne-grain host

memory accesses for sparse features while fully utilizing system in-

terconnect (e.g., PCIe) bandwidth, the proposed approach can oer

signicant advantage over the DMA approach. The removal of CPU

gathering stage not only shortens data access latency for GPUs,

but also greatly reduces the CPU and host memory utilization (Fig-

ure 3). Ooading CPU workloads to GPUs also helps on training

Contiguous Buffer

CPU

GPU

 DMA Read

 Write

DMA-based (Block Transfer) Approach

Zero-Copy-based Approach (This work)

Feature Tensor

CPU

GPU

Host

Feature Tensor

 Read

66.7% Memory

bandwidth saving

Lower end-to-end

data transfer latency

 Zero-Copy

Read

Busy

Idle

Figure 3: Workload comparison between DMA-based

method and the proposed zero-copy-based method.

GCN with multiple GPUs as we can prevent the CPU becoming the

bottleneck with increasing number of workers.

In order to propose the GPU-oriented data communication ar-

chitecture for GCN training, we address three major questions in

this work. First, can zero-copy memory access fully utilize PCIe

bandwidth while training GCN considering the long latency for

accessing host memory? Second, what would be the price of con-

suming GPU cores for zero-copy memory access? Finally, after

resolving the above two questions, can we show real end-to-end

application performance benet from our method?

In this work, we answer all three questions. First, to maintain

the best possible PCIe packet eciency with zero-copy memory

access, we propose an automatic data access alignment optimiza-

tion in GPU data indexing kernel. With our optimization, zero-copy

PCIe bandwidth can match up to 93% of block transfer PCIe band-

width. Second, we propose a novel CUDA multi-process service

(MPS) [

] based resource provisioning optimization to minimize

GPU resource consumption of zero-copy memory accesses. Based

on careful investigation of PCIe protocol and GPU architecture,

we conclude that we can saturate PCIe even if only a few number

of GPU cores are generating zero-copy accesses. Therefore, our

optimization isolates only small portion of GPU resources for the

zero-copy accesses and leaves the rest for computationally intense

workloads.

Finally, we build an end-to-end zero-copy GCN training ow in

PyTorch. To enable zero-copy memory access, we devise a new class

of tensor called "unied tensor". This tensor provides an address

mapping of host memory for GPUs so they can directly access host

memory with zero-copy accesses. By simply declaring multiple uni-

ed tensor instances for multiple GPUs, our GCN training ow can

also support zero-copy access in multi-GPU training environment.

2088

of 14

免费下载

paper vldb

关注

评论