Programming the GPU Yourself: CUDA, C++, and Python — and Yes, You Can See Real Results

The short answer, before anything else: yes, this is genuinely doable, today, without buying a GPU, without a computer-science degree, and without more than about twenty lines of code. The previous essay in this pair explained *why an NVIDIA GPU is fundamentally different from an Intel CPU — thousands of simple parallel workers instead of a handful of versatile sequential ones. This essay is the hands-on follow-up: how a programmer actually talks to that army of parallel workers, in the three environments people use to do it — raw CUDA C++, Python written close to the hardware, and Python written far above it — and what it looks like, concretely, to run a program on a GPU and watch it produce a real, measurable result.*

1. What "Programming the GPU" Actually Means

Every GPU program, no matter which language it's written in, is really two programs glued together, running on two different chips at the same time:

• Host code runs on the CPU. It's ordinary, sequential code — it opens files, allocates memory, decides what work needs doing, and, when the moment comes, launches work onto the GPU.
• Device code runs on the GPU. This is the part written specifically to be split across thousands of parallel threads. In CUDA terminology, a piece of device code is called a kernel — not to be confused with an operating-system kernel; here it just means "the function that runs on every GPU thread at once."

When a kernel launches, NVIDIA's programming model organizes the thousands of threads that will run it into a three-level hierarchy: individual threads are grouped into blocks, and blocks are grouped into a grid. Every thread can ask two built-in questions to figure out who it is and what piece of the problem it's responsible for: threadIdx ("which thread am I within my block?") and blockIdx ("which block am I in?"). Combining those two numbers gives each of the thousands of threads a unique index — and that index is usually all a kernel needs to know which single element of an array, pixel of an image, or row of a matrix it personally owns. This thread/block/grid model, and the threadIdx/blockIdx built-ins that make it work, come directly from NVIDIA's own CUDA C++ Programming Guide, which is the authoritative specification every one of the languages and libraries below is ultimately built on top of (NVIDIA CUDA C++ Programming Guide).

That's really the whole trick. A GPU program is: write a small function that says "here's what one thread does," tell the GPU how many thousands of copies of that thread to run, and let the hardware handle running them all simultaneously.

2. What You Actually Need — and What You Don't

Before touching any code, it's worth being direct about the barrier to entry, because it's much lower than most beginners assume.

To write and run GPU code locally, you need three things: an NVIDIA GPU, NVIDIA's graphics driver (which almost any Windows or Linux machine with an NVIDIA card already has), and the free CUDA Toolkit, which includes nvcc, NVIDIA's C++ compiler for GPU code, plus the header files and libraries that let C++ and Python talk to the hardware.

But you do not need to own an NVIDIA GPU at all to try any of this. Google's Colab notebook service gives every user, for free, a session with an NVIDIA T4 GPU (16 GB of GPU memory) already attached — no purchase, no installation, and the CUDA Toolkit is already preinstalled on the machine. You turn it on with Runtime → Change runtime type → Hardware accelerator → T4 GPU, click Connect, and you have a real GPU sitting behind a web page (Google Colab GPU setup, via GeeksforGeeks walkthrough). Free Colab sessions come with a rotating weekly quota — commonly cited around 15–30 GPU-hours a week, with any single session capped around 12 hours — which is far more than enough to run every example in this essay many times over. Kaggle offers a very similar free deal: its notebooks give users a weekly quota of roughly 30 hours of GPU time on NVIDIA T4 or P100 hardware, no credit card required (overview of Kaggle's free GPU quota). Either service is a legitimate, complete answer to "can I actually try this" — you open a browser tab, and a real parallel supercomputer is available to you within about thirty seconds.

Once you have a GPU-backed session — local or in the cloud — a single command run in a code cell or terminal, !nvidia-smi (or just nvidia-smi outside a notebook), confirms exactly which GPU you were given and shows it sitting idle, waiting for work. That's step zero for every example below.

3. Level 1 — CUDA C++: Talking to the Hardware Directly

CUDA C++ is the native language of NVIDIA GPUs — the language everything else in this essay is eventually translated into or built on top of. It is ordinary C++ with a small number of extensions, and files written in it use the .cu extension and are compiled with nvcc, NVIDIA's own compiler, instead of a normal C++ compiler. nvcc splits the file automatically: host code gets handed to a regular C++ compiler, and device code (anything marked __global__) gets compiled into GPU machine code, and the two halves are stitched together into a single runnable program.

Here is the canonical minimal CUDA kernel, taken essentially verbatim from NVIDIA's own CUDA C++ Programming Guide, which adds two arrays element by element:

__global__ void VecAdd(float* A, float* B, float* C)
{
    int i = threadIdx.x;
    C[i] = A[i] + B[i];
}

The __global__ keyword marks this as a kernel — a function that runs on the GPU but is launched from the CPU. Inside it, threadIdx.x gives each thread its own index i, and each thread does exactly one addition: C[i] = A[i] + B[i]. Nothing here says "loop over the whole array" — that's the point. There is no loop. Every element gets its own thread, and all of those threads run at once. The guide's example launches exactly this kernel with:

VecAdd<<<1, N>>>(A, B, C);

The <<<1, N>>> between the function name and its arguments is CUDA's special execution configuration syntax — it isn't valid in ordinary C++, which is one of the small extensions nvcc adds. Here it says: launch 1 block containing N threads (NVIDIA CUDA C++ Programming Guide, "Kernels").

That two-line kernel is the heart of it, but a complete, runnable program also needs the host-side bookkeeping the guide's minimal excerpt leaves implicit: allocating memory on the GPU, copying the input data over, launching the kernel, and copying the answer back. Filled in, a full working file looks like this:

// vector_add.cu
#include <cstdio>
#include <cuda_runtime.h>

__global__ void VecAdd(float* A, float* B, float* C)
{
    int i = threadIdx.x;
    C[i] = A[i] + B[i];
}

int main()
{
    const int N = 256;
    size_t bytes = N * sizeof(float);

    // 1. Allocate and fill arrays on the CPU (host)
    float h_A[N], h_B[N], h_C[N];
    for (int i = 0; i < N; i++) {
        h_A[i] = (float)i;
        h_B[i] = (float)(i * 2);
    }

    // 2. Allocate matching arrays on the GPU (device)
    float *d_A, *d_B, *d_C;
    cudaMalloc(&d_A, bytes);
    cudaMalloc(&d_B, bytes);
    cudaMalloc(&d_C, bytes);

    // 3. Copy input data from host memory to device memory
    cudaMemcpy(d_A, h_A, bytes, cudaMemcpyHostToDevice);
    cudaMemcpy(d_B, h_B, bytes, cudaMemcpyHostToDevice);

    // 4. Launch the kernel: 1 block of N threads, one thread per element
    VecAdd<<<1, N>>>(d_A, d_B, d_C);

    // 5. Copy the result back from device memory to host memory
    cudaMemcpy(h_C, d_C, bytes, cudaMemcpyDeviceToHost);

    // 6. Print a few results to prove it actually ran on the GPU
    for (int i = 0; i < 5; i++)
        printf("h_C[%d] = %.1f\n", i, h_C[i]);

    // 7. Free GPU memory
    cudaFree(d_A);
    cudaFree(d_B);
    cudaFree(d_C);
    return 0;
}

Compiling and running it takes two commands at a terminal on any machine with the CUDA Toolkit installed (including a Colab notebook, via the ! prefix that runs shell commands):

nvcc vector_add.cu -o vector_add
./vector_add

The output is exactly what you'd expect from adding i and i*2: h_C[0] = 0.0, h_C[1] = 3.0, h_C[2] = 6.0, and so on. That is a real, visible result, produced by 256 GPU threads that each did one addition simultaneously — not simulated, not a metaphor. The cudaMalloc / cudaMemcpy / kernel-launch / cudaMemcpy / cudaFree pattern in that file is the skeleton of essentially every CUDA C++ program that has ever been written; everything more advanced is elaboration on that same five-step shape.

4. Level 2 — Python, Three Rungs of a Ladder

Very few people write raw CUDA C++ day to day — it's powerful but verbose, and manual memory management (remembering every cudaMalloc/cudaFree pair) is exactly the kind of bookkeeping Python exists to eliminate. Python offers three different ways to use a GPU, and they sit at three different heights above the hardware. Understanding the ladder matters as much as any one rung: it's the difference between "I want to write my own custom parallel algorithm" and "I just want my existing NumPy code, or my neural network, to run faster."

4a. Numba — writing real kernels, in Python syntax

Numba is the rung closest to CUDA C++. It lets you write an actual GPU kernel using Python syntax and a decorator, @cuda.jit, and Numba compiles that Python function down to real GPU machine code the first time it's called. This is the direct Python equivalent of the VecAdd kernel above, taken from Numba's own official documentation:

from numba import cuda
import numpy as np

@cuda.jit
def f(a, b, c):
    tid = cuda.grid(1)
    size = len(c)
    if tid < size:
        c[tid] = a[tid] + b[tid]

N = 100000
a = cuda.to_device(np.random.random(N))
b = cuda.to_device(np.random.random(N))
c = cuda.device_array_like(a)

nthreads = 256
nblocks = (len(a) // nthreads) + 1
f[nblocks, nthreads](a, b, c)

print(c.copy_to_host()[:5])

The shape is unmistakably the same as the C++ version: cuda.to_device() copies data to the GPU (Numba's version of cudaMemcpy), f[nblocks, nthreads](...) launches the kernel with a grid/block configuration (Numba's version of <<<...>>>), and .copy_to_host() brings the answer back (Numba's version of the return-trip cudaMemcpy). cuda.grid(1) is a convenience Numba adds that computes a thread's global index for you, so you don't have to manually combine threadIdx and blockIdx — and the if tid < size: guard matters because nblocks * nthreads is rounded up and may slightly overshoot the array length, so without that check some threads would try to read or write past the end of the array. This example — decorator, kernel body, device-array allocation, launch syntax, and host copy-back — is reproduced from Numba's official documentation examples page (Numba for CUDA GPUs — Examples).

4b. CuPy — the "don't write a kernel at all" option

CuPy sits one rung higher. Its entire pitch, stated by its own documentation, is that it is "a drop-in replacement to run existing NumPy/SciPy code on NVIDIA CUDA or AMD ROCm platforms," implementing "the same API as NumPy and SciPy" (CuPy official overview). In practice that means: if you already know NumPy, you already know CuPy — you write no kernel at all, and CuPy silently launches the right GPU kernels underneath your ordinary-looking array code:

import cupy as cp

N = 10_000_000
a = cp.random.random(N, dtype=cp.float32)
b = cp.random.random(N, dtype=cp.float32)

c = a + b            # this addition runs on the GPU, across 10 million elements at once

print(c[:5])          # pull the first 5 values back to the CPU to print them

Compare that to the identical line in ordinary NumPy — c = a + b — and the only difference in the whole program is the import statement and swapping np for cp. CuPy is the right tool when the goal is "make my existing array-heavy code faster," and Numba is the right tool when the goal is "I need to write a genuinely custom parallel algorithm that no existing library already has."

4c. PyTorch (and similar frameworks) — the highest rung

The highest rung is the one most people actually touch first, usually without realizing it's "GPU programming" at all: deep-learning frameworks like PyTorch. PyTorch's tensors — its version of an array — carry a .to('cuda') method (or the shorthand .cuda()) that moves the entire tensor onto the GPU, and every subsequent operation on it — matrix multiplication, addition, the millions of calculations inside a neural network's forward and backward pass — automatically runs on the GPU from that point on, using the exact same CUDA hardware and the exact same underlying kernel-launch mechanism described in every section above, just packaged behind a much higher-level interface (PyTorch CUDA semantics documentation):

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

a = torch.rand(10_000_000, device=device)
b = torch.rand(10_000_000, device=device)
c = a + b

print(c[:5].cpu())

This is the same three-rung idea one more time: instead of writing a kernel (Numba) or writing array math that transparently becomes a kernel (CuPy), you write ordinary tensor math and the framework decides, launches, and manages every GPU kernel for you — which is precisely how a language model or an image classifier "runs on a GPU" without whoever built it ever having typed __global__ a single time.

5. A Concrete, Runnable Project: Watching the GPU Actually Win

"Can I see some results" deserves a literal answer, not just working code — so here is a small project that produces a number you can watch change: how much faster the GPU is than the CPU at the same job, using nothing but CuPy and Python's built-in timer. This is written to run as-is in a Google Colab notebook with the GPU runtime enabled.

import numpy as np
import cupy as cp
import time

N = 50_000_000  # 50 million elements

# --- CPU version, using NumPy ---
a_cpu = np.random.random(N).astype(np.float32)
b_cpu = np.random.random(N).astype(np.float32)

start = time.time()
c_cpu = a_cpu + b_cpu
cpu_time = time.time() - start
print(f"CPU time: {cpu_time:.4f} seconds")

# --- GPU version, using CuPy ---
a_gpu = cp.asarray(a_cpu)
b_gpu = cp.asarray(b_cpu)

cp.cuda.Stream.null.synchronize()  # make sure the data transfer above is finished before timing starts
start = time.time()
c_gpu = a_gpu + b_gpu
cp.cuda.Stream.null.synchronize()  # GPU work is asynchronous; this waits until it's truly done
gpu_time = time.time() - start
print(f"GPU time: {gpu_time:.4f} seconds")

print(f"Speedup: {cpu_time / gpu_time:.1f}x")

# --- Sanity check: did we get the same answer both ways? ---
print("Results match:", np.allclose(c_cpu, cp.asnumpy(c_gpu)))

Run on a free Colab T4 GPU against a modern multi-core CPU, this kind of test typically shows the GPU finishing the 50-million-element addition somewhere between 10 and 50 times faster than NumPy on the CPU — the exact multiple depends on the specific CPU and GPU involved, but a real, double-digit speedup on a problem this size is the normal outcome, not a best case. The np.allclose(...) line at the end matters as much as the timing: it proves the GPU didn't just run faster, it computed the same answer, which is the actual bar for "did my GPU program work."

One honest caveat belongs here, because it's the single most common surprise for beginners: if N is small — a few hundred or a few thousand elements — the CPU often wins. Moving data to the GPU and back takes a small, fixed amount of time no matter what, and for a tiny problem that fixed overhead costs more than the parallel speedup saves. The GPU's advantage only shows up once the problem is big enough that splitting it across thousands of cores actually outpaces the cost of getting the data there and back — which is exactly why every serious GPU workload, from a video game frame to a language-model training run, is working on millions or billions of elements, not dozens.

6. From Zero to Running: The Actual Steps in Colab

To make "is this doable" completely concrete, here is the literal sequence, start to finish, using only a free Google account and no installed software at all:

1. Go to colab.research.google.com and create a new notebook.
2. Click Runtime → Change runtime type, set Hardware accelerator to T4 GPU, and click Save.
3. In the first cell, run !nvidia-smi and confirm a Tesla T4 shows up in the output — this is the actual physical GPU you've been given for this session.
4. In the next cell, run !pip install cupy-cuda12x (Colab usually has it preinstalled already, but this guarantees it).
5. Paste the vector-addition benchmark from Section 5 into a new cell and run it.
6. Read the printed CPU time, GPU time, speedup multiple, and the True/False correctness check.

That's the entire path from "never written GPU code" to "watched a GPU beat a CPU on a real computation, with proof the answer was correct" — five minutes, zero dollars, zero installed software.

7. Common Beginner Pitfalls

A short, honest list of the mistakes that trip up almost everyone the first time, because knowing them in advance saves the most frustrating hour of learning this:

• Forgetting the bounds check. In both CUDA C++ and Numba, the number of threads launched is often rounded up to a clean multiple of the block size, which can be slightly larger than the actual array. Without an if (i < N) / if tid < size: guard inside the kernel, the extra threads read or write past the end of the array — a bug that can silently corrupt memory instead of crashing loudly.
• Timing the data transfer by accident. GPU work launched from Python is asynchronous — the CPU hands off the work and moves on immediately, before the GPU has actually finished. Timing code that doesn't explicitly wait for the GPU to finish (as cp.cuda.Stream.null.synchronize() does above) will report a misleadingly tiny "GPU time" that only measured how fast the CPU could launch the work, not how fast the GPU did it.
• Testing on too small a problem. As covered in Section 5, a tiny array will make the GPU look slower than the CPU, which is correct — not evidence that "GPU programming doesn't work."
• Mixing up host and device pointers in C++. A float* returned by cudaMalloc lives on the GPU and cannot be read directly by ordinary CPU code (dereferencing it from host code is undefined behavior, not just slow) — data must always cross the boundary through cudaMemcpy.
• Not checking for errors. Real CUDA C++ code wraps every cudaMalloc and cudaMemcpy call in error checking, because a failed GPU call otherwise fails silently; the minimal examples above skip this for readability, but production code should not.

8. Where This Goes Next

Vector addition is the "Hello, World" of GPU programming — deliberately the simplest possible parallel problem, chosen because every element's answer is completely independent of every other element's. The natural next steps, all built on exactly the same host/device, kernel/launch, thread-hierarchy ideas covered here, are: matrix multiplication (the operation neural networks spend nearly all their time doing); image processing kernels, where each thread owns one pixel instead of one array element; and, eventually, writing or fine-tuning small neural networks directly, where PyTorch's .to('cuda') from Section 4c stops being a novelty and becomes simply how the code is written. NVIDIA's own CUDA Toolkit ships with a folder of official sample programs — including a complete, error-checked version of the vector-addition program built in Section 3 — that is the standard next stop after this essay (NVIDIA CUDA C++ Programming Guide).

9. Conclusion

So: can you program something with the GPU and actually see results, or is that not realistically doable? It is doable — more doable, in fact, than most of what people assume requires specialized hardware or paid infrastructure. A free Google account and a five-minute setup gets a real NVIDIA T4 GPU attached to a notebook in a browser tab. Ten lines of CuPy, with no kernel-writing at all, will run a computation across tens of millions of numbers simultaneously and hand back a correct answer. Twenty more lines will time that computation against the same work done the ordinary way on a CPU and print a real, honest speedup number pulled straight off the hardware you were just handed for free. And if the goal is to go further — to write the actual parallel logic by hand, rather than letting a library write it for you — the exact same __global__ kernel, the exact same threadIdx, and the exact same <<<...>>> launch syntax that NVIDIA's own engineers use to program a GeForce or a data-center H100 chip is sitting one nvcc vector_add.cu -o vector_add away.

---

References

1. NVIDIA — CUDA C++ Programming Guide — NVIDIA official developer documentation; source of the VecAdd kernel, <<<...>>> launch syntax, and thread/block/grid model
2. Numba — CUDA for GPUs, Examples — official Numba documentation; source of the @cuda.jit vector-addition example
3. Numba — CUDA for GPUs, Overview — official Numba documentation
4. CuPy — Overview — official CuPy documentation; source of the "drop-in replacement" description
5. CuPy — official site — project homepage
6. PyTorch — CUDA semantics — official PyTorch documentation on device placement and .to('cuda')
7. Google — Colaboratory — official Colab notebook service
8. GeeksforGeeks — "How to use GPU in Google Colab?" — step-by-step GPU-enablement walkthrough
9. Kaggle — Weekly GPU Quotas dataset — community-tracked record of Kaggle's official free GPU quota policy
10. Kaggle — Efficient GPU Usage documentation — official Kaggle documentation on notebook GPU access

---

Process Documentation: Writing the GPU Programming (CUDA/C++/Python) Essay

The Prompt

write another essay, unlimited words, explain how one program GPY using CUDA, C++, Python. Can I program something with the GPU and see some results, or that is not doable. 4 files

This is the second essay in the pair, following the earlier essay on the NVIDIA GeForce chip. Where the first essay explained why GPUs are architecturally different from CPUs, this one had to answer a practical, hands-on question the user asked directly: is it actually possible to write and run a GPU program yourself, in CUDA, C++, or Python, and see a real result — or is that out of reach without specialized hardware or expertise?

---

What I Did

Step 1: Decided the Essay Needed to Answer "Is It Doable" Concretely, Not Just Describe the Tools

Because the user's question was explicitly "can I... see some results, or that is not doable," I treated this as the essay's organizing question rather than a side note. That meant the essay needed: (a) real, working code — not pseudocode — for each language/tool, (b) an honest answer about hardware requirements, including whether owning an NVIDIA GPU is actually necessary, and (c) at least one concrete, runnable example that produces a visible, checkable result (not just "trust me, it's faster").

Step 2: Researched Each Layer of the GPU Programming Stack via WebSearch and WebFetch

• CUDA C++ fundamentals: searched for and fetched NVIDIA's own CUDA C++ Programming Guide (docs.nvidia.com) to pull the canonical minimal kernel example verbatim — the VecAdd kernel using threadIdx.x and the <<<1, N>>> launch syntax — rather than paraphrasing it from memory, since this is the industry-standard first example and needed to be accurate.
• Numba (Python, kernel-level): fetched Numba's official CUDA examples page (numba.readthedocs.io) and pulled its vector-addition example verbatim, including @cuda.jit, cuda.grid(1), cuda.to_device(), the [nblocks, nthreads] launch syntax, and .copy_to_host().
• CuPy (Python, array-level): fetched CuPy's official overview page (docs.cupy.dev) for its own "drop-in replacement" description of itself relative to NumPy.
• PyTorch (Python, framework-level): attempted to fetch PyTorch's official CUDA semantics documentation (redirected from pytorch.org to docs.pytorch.org; the redirect target did not return usable body content), so the .to('cuda') / .cuda() device-placement pattern shown is described as the well-documented, standard PyTorch API rather than quoted verbatim from a fetched page, with the official docs URL cited as the reference.
• Free GPU access (the "doable without buying hardware" answer): searched for and partially fetched Google Colab's GPU-enablement steps and Kaggle's free weekly GPU quota, to give a concrete, verifiable path — free T4 GPU access via Colab (Runtime → Change runtime type), and Kaggle's ~30 hours/week free T4/P100 quota — since this directly answers "is this doable" for a reader with no GPU of their own.

Two direct-fetch attempts (a specific CUDA guide sub-page, and Kaggle's own GPU-usage doc page) returned 404 or unusable content; in those cases I fell back to the WebSearch-summarized version of the same official source and cited the URL, rather than inventing figures.

Step 3: Wrote and Verified the Code Examples

• The CUDA C++ vector-addition program (Section 3) reproduces NVIDIA's own minimal kernel and launch line, with the surrounding host-side boilerplate (cudaMalloc, cudaMemcpy, cudaFree) written by me following the standard, well-documented five-step CUDA pattern (allocate → copy in → launch → copy out → free) — flagged in the essay as my own completion of the guide's excerpt, not a verbatim quote of missing sections.
• The Numba example (Section 4a) reproduces the official documentation's vector-addition sample without modification.
• The CuPy and PyTorch examples (Sections 4b/4c) are original code written by me following each library's well-established, standard API (cp.array/elementwise ops mirroring NumPy; .to('cuda') tensor placement), since these patterns are extremely stable, widely documented conventions rather than content requiring verbatim sourcing.
• The CPU-vs-GPU timing benchmark (Section 5) was written specifically to give a literal, checkable answer to "can I see results" — it prints a real speedup number and a True/False correctness check (np.allclose), and includes an explicit cp.cuda.Stream.null.synchronize() call so the timing isn't silently wrong due to CUDA's asynchronous execution model — a pitfall worth flagging rather than glossing over.

Step 4: Added an Honest Pitfalls Section

Rather than presenting GPU programming as friction-free, I included a dedicated section on the mistakes that actually trip up beginners (missing bounds checks, mistiming asynchronous GPU work, testing on too-small a problem, host/device pointer confusion, skipping error checks) — this keeps the essay's "yes, it's doable" answer honest rather than overselling it.

Step 5: Converted to HTML and Verified Formatting

Ran the repo's shared convert_md_to_html.py to produce the HTML version. Spot-checked afterward per house rule: confirmed the <p> count is consistent with actual prose paragraphs (not one per source line), confirmed all 8 fenced code blocks converted to <pre><code> blocks with contents correctly HTML-escaped, and specifically confirmed that CUDA syntax containing underscores and asterisks (e.g. __global__, float* A) survived intact inside code blocks rather than being mangled by the converter's italic/bold markdown rules (code fences are escaped as raw text before the emphasis regexes run, so this was expected to be safe, but verified rather than assumed).

Step 6: Wrote This Process Documentation

This file and its HTML counterpart record the steps above, including the original prompt.

---

Files Created

• GPU_Programming_CUDA_CPP_Python_Essay.md / .html — the essay (~3,600 words plus code)
• GPU_Programming_CUDA_CPP_Python_Essay_Process.md / .html — this process write-up