Published 2026-10-19.
Time to read: 4 minutes.
llm collection.
Local LLMs are private, unlike cloud-based models, and they do not require a subscription or account to use.
This article demonstrates how I installed Gemma-4 for an NIVIDIA RTX 3060 and used Ollama to run it with Visual Studio Code on Windows. This article shows how to configure VS Code to use Ollama's local LLMs with the built-in chat and coding tools.
This article also discusses how to configure Visual Studio Code to access Ollama models on the local machine, and on another machine on the same network. This allows you to run Gemma-4 on a powerful machine with an NVIDIA RTX 4090 or A100, and efficiently access it from a less powerful machine that does not have a GPU suitable for running the model.
BYOK
Visual Studio Code now has a native "Bring Your Own Key " (BYOK) model architecture and Agent Harness system built right into the core editor.
You no longer have to use complex third-party extensions like Continue if you just want to use VS Code's built-in chat window and coding tools with local models.
The built-in VS Code agent harness handles vision capabilities as well, meaning your Gemma-4 configuration can parse highlighted code snippets and UI layouts directly inside the built-in chat viewport.
Gemma-4 Versions
All Gemma-4 versions provide native audio and image support.
Gemma 4 E2B
- Parameters: 2.3B effective (5.1B with embeddings)
- Context Window: 128K tokens
- Smallest and fastest version. Built for mobile, edge, and browser deployment.
Gemma 4 E4B
- Parameters: 4.5B effective (8B with embeddings)
- Context Window: 128K tokens
- Designed for offline edge processing on mobile, IoT devices, and laptops.
Gemma 4 12B Unified
- Parameters: 11.95B dense, encoder-free
- Context Window: 256K tokens
- Leaves over 5 GB of headroom on an RTX 3060 for long context windows and KV cache.
Gemma 4 26B A4B
Not suitable for consumer-grade GPUs, but can be run on high-end NVIDIA RTX 4090 or A100 GPUs with 24GB of VRAM. It is designed for advanced reasoning, coding, and multimodal tasks.
- Parameters: 26B total parameters with 4B active parameters per token
- Context Window: 256K tokens
- High-throughput Mixture-of-Experts (MoE) model balancing fast inference speeds with strong reasoning and coding performance.
Gemma 4 31B
Not suitable for consumer-grade GPUs, but can be run on high-end NVIDIA RTX 4090 or A100 GPUs with 24GB of VRAM. It is the most capable model in the Gemma 4 lineup, and is designed for advanced reasoning, coding, and multimodal tasks.
- Parameters: 31B dense model
- Context Window: 256K tokens
- Flagship high-performance dense model serving as the most capable reasoner in the Gemma 4 lineup.
Installation
Install Ollama
The following command installs or updates the latest version of Ollama on Bash.
$ curl -fsSL https://ollama.com/install.sh | sh >>> Cleaning up old version at /usr/local/lib/ollama >>> Installing ollama to /usr/local >>> Downloading ollama-linux-amd64.tar.zst ######################################################### 100.0% >>> Adding ollama user to render group... >>> Adding ollama user to video group... >>> Adding current user to ollama group... >>> Creating ollama systemd service... >>> Enabling and starting ollama service... >>> NVIDIA GPU installed.
Installing Gemma-4
The E2B, E4B, and 12B versions of
Gemma-4 work on the NVIDIA RTX 3060, but the
12B model is the most capable. The following commands show how to
download each version of Gemma-4 to the local machine. Install the version(s)
that best fit your needs.
$ ollama pull gemma4:12b pulling manifest pulling 0cb0cd0a5c49: 100% ▕██████████████████████████████████████▏ 175 MB pulling bb722270d543: 100% ▕██████████████████████████████████████▏ 7.4 GB pulling 7008a656050b: 100% ▕██████████████████████████████████████▏ 465 MB pulling b507b9c2f6ca: 100% ▕██████████████████████████████████████▏ 13 B pulling 0d542e0c8804: 100% ▕██████████████████████████████████████▏ 10 KB pulling 2365fbb6d97b: 100% ▕██████████████████████████████████████▏ 64 B pulling e5ca99f91896: 100% ▕██████████████████████████████████████▏ 242 B verifying sha256 digest writing manifest removing unused layers success
The E4B model is faster. The following command downloads it to
the local machine:
$ ollama pull gemma4:e4b pulling manifest pulling 370c2879f176: 100% ▕██████████████████████████████████▏ 5.5 GB/5.5 GB 105 MB/s 0s pulling 4cb21b935cd7: 100% ▕██████████████████████████████████▏ 991 MB/991 MB 107 MB/s 0s pulling 05612e54110f: 100% ▕██████████████████████████████████▏ 98 MB/ 98 MB 44 MB/s 0s pulling b507b9c2f6ca: 100% ▕██████████████████████████████████▏ 13 B pulling 2365fbb6d97b: 100% ▕██████████████████████████████████▏ 64 B pulling 7ca1ae564b24: 100% ▕██████████████████████████████████▏ 241 B verifying sha256 digest writing manifest success
The E2B model is the fastest. The following command downloads it to
the local machine:
$ ollama pull gemma4:e2b pulling manifest pulling 6dd7ac24fe13: 100% ▕██████████████████████████████████ ▏ 3.5 GB/3.5 GB 107 MB/s 0s pulling 7ba3bbacdf1b: 100% ▕██████████████████████████████████ ▏ 983 MB/986 MB 107 MB/s 0s pulling dd57b3b7308f: 100% ▕███████████████████████████████████▏ 97 MB pulling b507b9c2f6ca: 100% ▕███████████████████████████████████▏ 13 B pulling 7339fa418c9a: 100% ▕███████████████████████████████████▏ 11 KB pulling 2365fbb6d97b: 100% ▕███████████████████████████████████▏ 64 B pulling 3ff8a734c316: 100% ▕███████████████████████████████████▏ 241 B verifying sha256 digest writing manifest success
Install Ollama Extension For VSCode
The documentation pushes you towards installing a Visual Studio Code extension. This is not necessary, and using the GUI for configurion is confusing, but perhaps this will improve over time. The Configuration section discusses how to configure Ollama for Visual Studio Code without the extension.
You can install the Ollama extension for Visual Studio Code from the Visual Studio Code Marketplace or by running the following command:
$ code --install-extension ollama.ollama Installing extension 'ollama.ollama'... Extension 'ollama.ollama' v0.0.13 was successfully installed.
Configuration
You have two ways to connect Ollama to the new Visual Studio Code built-in harness: via an extension or by manually creating a config file.
Method 1: Ollama Marketplace Extension
VS Code natively supports provider decoupling, and the official recommendation for local Ollama models is to use the dedicated Ollama Extension on the Visual Studio Marketplace. This hooks Ollama straight into VS Code’s built-in chat infrastructure.
I found this method to be confusing because Visual Studio Code’s GUI has been changing on every release. Documentation does not match the current GUI, and the extension’s documentation is not very clear. The following steps worked for me to get the Ollama extension working with Gemma-4 on a local GPU.
-
Requires VS Code 1.127 or newer.
-
Install the official Ollama extension from the Marketplace as described earlier.
-
Open the built-in VS Code Chat panel (CTRL + ALT + I or CMD + SHIFT + I).
-
Click the Model Picker dropdown at the bottom of the chat panel input box.
-
You will see a dedicated Ollama section automatically listing the models you have downloaded locally. Select it and start coding.
Method 2: Define Custom Endpoint
You do not need to use a Visual Studio Extension. To map the native chat engine manually to your local Ollama port, you can use the built-in custom endpoints feature.
-
Open the Command Palette (CTRL + SHIFT + P or CMD + SHIFT + P).
-
Type and select Chat: Manage Language Models.
-
Add a Custom Endpoint.
-
Point it to the OpenAI-compatible completions endpoint for the computer that has the GPU. In this example, network node
gojirahas the GPU:-
Model ID:
gemma4:12b
-
Save the configuration. This model will instantly show up as a target inside the main VS Code UI without requiring any active GitHub Copilot subscriptions or cloud logins.
