NuGet without Internet Survival Guide

Technicaly we can work normally without Internet by preparing required Docker images for storages like ElasticSearch and SqlServer

But when Internet dissapear and you switch between branches, Rider may decide to refresh packages and immediately will dia with bunch of errors about broken dependencies

To avoid such we gonna need to have local NuGet kind of cache

Thankfully NuGet has build in feature exactly for such scenario

All you need to do:

save wanted packages somewhere on local machine
add source pointing to folder where you saved all of them

Here is an short demo of how it can be used:

echo 'recreate offline directory for demo'
rm -rf offline || true
mkdir offline

echo 'create demo classlib'
rm -rf mylib || true
mkdir mylib
cd mylib
dotnet new console
dotnet pack
cd ..
echo 'pretend like we downloaded some nupkg for offline usage'
mv mylib/bin/Debug/mylib.1.0.0.nupkg ./offline/

echo 'list current nuget sources'
dotnet nuget list source
echo 'remove offline source if any'
dotnet nuget remove source offline
echo 'add offline source'
dotnet nuget add source "${PWD}/offline" --name=offline


echo 'remove original nuget.org source'
dotnet nuget remove source nuget.org

echo 'demo of offline usage'
rm -rf consoleapp || true
mkdir consoleapp
cd consoleapp
dotnet new console
dotnet add package mylib
dotnet restore
cd ..

echo 'add nuget.org back' # TODO: not sure if thats really important but given command does not add 'protocolVersion="3"', aka: '<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />'
dotnet nuget add source https://api.nuget.org/v3/index.json --name=nuget.org

echo 'cleanup'
rm -rf consoleapp mylib offline || true

Now we need to collect all packages we are using:

$basedir = '/path/to/dir/where/you/store/projects'
$repos = Get-ChildItem -Path $basedir -Directory | Select-Object -ExpandProperty Name
$packages = @()
foreach($repo in $repos) {
    $projects = Get-ChildItem -Path (Join-Path $basedir $repo) -Recurse -Depth 3 -Include '*.csproj'
    foreach($project in $projects) {
        $xml = [xml](Get-Content $project.FullName)
        foreach($reference in $xml.Project.ItemGroup.PackageReference) {
            if (-not $reference) { continue }
            $packages += [PSCustomObject]@{
                Repository = $repo
                Project = [System.IO.Path]::GetFileNameWithoutExtension($project.FullName)
                Package = $reference.Include
                Version = $reference.Version
            }
        }
    }
}

$packages | ConvertTo-Json | Out-File -Encoding utf8 packages.json

This script will produce JSON with entries like:

{
  "Repository": "MyAwesomeApp",
  "Project": "MyAwesomeApp",
  "Package": "Newtonsoft.JSON",
  "Version": "12.0.3"
}

Now, when we will have an Internet, we may download them with URL like:

Invoke-WebRequest "https://api.nuget.org/v3-flatcontainer/$package/$version/$package.$version.nupkg" -OutFile "/Users/mac/Downloads/nucache/$package.$version.nupkg"

But, make sure to download package dependencies as well, otherwise when you will be offline an surprise is waiting for you, you gonna need not only download all that fancy libraries but bunch of system and Microsoft libraries as well

TODO: script out dependencies download

The best part of this approach is that you are downloading everything you need upfront which is much better than using any caching proxies

Alternative approaches

My very first attempt was to run nginx as a caching proxy

Here is config I ended up with:

# https://www.nginx.com/blog/nginx-caching-guide/
# http://nginx.org/en/docs/http/ngx_http_proxy_module.html
# https://gist.github.com/sirsquidness/710bc76d7bbc734c7a3ff69c6b8ff591
proxy_cache_path /cache keys_zone=nucache:10m inactive=100d max_size=30g;

server {
    listen       80;
    server_name  localhost;

    location / {
        proxy_cache nucache;

        proxy_pass http://demoa;
        # replace with ip address because when there is no internet, address may be not resolved
        # proxy_pass https://api.nuget.org;
        # proxy_pass https://152.199.23.209;
        # proxy_set_header Host api.nuget.org;
        # better option is to add: "--add-host api.nuget.org:152.199.23.209" to docker run command

        # Delivering Cached Content When the Origin is Down
        proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504 http_403 http_404 http_429;
        # This one should be added/used only when we have internet, in combination with "updating" option in "proxy_cache_use_stale" nginx will serve cached file, but in background will try to update it
        # proxy_cache_background_update on;

        proxy_cache_key $scheme$proxy_host$uri$is_args$args;
        proxy_cache_valid 48h;
        proxy_cache_methods GET HEAD POST;
        add_header X-Nuget-Cache $upstream_cache_status;

        proxy_intercept_errors on;
        error_page 301 302 307 = @handle_redirects;

        proxy_ignore_headers Set-Cookie;
        proxy_ignore_headers Cache-Control;
        proxy_ignore_headers Expires;
        proxy_hide_header Set-Cookie;
        proxy_hide_header Cache-Control;
        proxy_hide_header Expires;
    }

    location @handle_redirects {
        # store the current state of the world so we can reuse it in a minute
        # We need to capture these values now, because as soon as we invoke
        # the proxy_* directives, these will disappear
        # set $original_uri $uri;
        set $original_scheme $scheme;
        set $original_proxy_host $proxy_host;
        set $original_uri $uri;
        set $original_is_args $is_args;
        set $original_args $args;

        set $orig_loc $upstream_http_location;

        # nginx goes to fetch the value from the upstream Location header
        proxy_pass $orig_loc;
        proxy_cache nucache;
        # But we store the result with the cache key of the original request URI
        # so that future clients don't need to follow the redirect too
        # proxy_cache_key $original_uri;
        proxy_cache_key $original_scheme$original_proxy_host$original_uri$original_is_args$original_args;
        proxy_cache_valid 48h;
        proxy_cache_methods GET HEAD POST;

        # resolver 8.8.8.8; # Seems to be needed for redirects.
        proxy_ignore_headers Set-Cookie;
        proxy_ignore_headers Cache-Control;
        proxy_ignore_headers Expires;
        proxy_hide_header Set-Cookie;
        proxy_hide_header Cache-Control;
        proxy_hide_header Expires;
    }
}

But it did not worked out as expected, having troubles with redirections and cache keys, somehow it does not work and it is not easy to debug all this, plus if we want to cache requests with body there will be even more troubles, so it seems to be not working solution

The second attempt was to build own nano proxy exactly for nuget

Here is what I have ended up with:

package main

import (
	"bytes"
	"crypto/sha512"
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
	"net/url"
	"os"
	"path"
)

var online bool
var target string
var cache string
var port int

type CacheRequest struct {
	Method string
	Path   string
	Body   []byte
}

type CacheResponse struct {
	Status int
	Header http.Header
	Body   []byte
}

type CacheEntry struct {
	Request  CacheRequest
	Response CacheResponse
}

func main() {
	flag.StringVar(&target, "target", "https://api.nuget.org", "proxy target")
	flag.StringVar(&cache, "cache", "./cache", "cache directory")
	flag.BoolVar(&online, "online", true, "online mode")
	flag.IntVar(&port, "port", 8080, "port to listen")
	flag.Parse()

	fmt.Println("nucache")
	fmt.Println()
	fmt.Println("target:", target)
	fmt.Println("cache:", cache)
	fmt.Println("online:", online)
	fmt.Println("port:", port)
	fmt.Println()
	fmt.Println("commands to switch between proxy and nuget:")
	fmt.Println("  dotnet nuget list source")
	fmt.Println("  dotnet nuget remove source nuget.org")
	fmt.Printf("  dotnet nuget add source http://localhost:%d/v3/index.json --name=nuget.org\n", port)
	fmt.Println("  dotnet nuget add source https://api.nuget.org/v3/index.json --name=nuget.org")
	fmt.Println()
	fmt.Println("commands to switch online/offline:")
	fmt.Printf("  curl http://localhost:%d/online\n", port)
	fmt.Printf("  curl http://localhost:%d/offline\n", port)
	fmt.Println()
	fmt.Println("notes:")
	fmt.Println("- in online mode cache is not used at all, but all ok'ish responses are saved")
	fmt.Println("- in offline mode network is not used at all, only saved cache entries are used")
	fmt.Println("- switch between modes is manual")
	fmt.Println("- there is no concept of time to live, cache entries live forever")
	fmt.Println("- to renew cache entry just switch to online mode")
	fmt.Println("- to remove cache entry just delete its file")
	fmt.Println()
	fmt.Println("example of online log:")
	fmt.Println("Status: online, Method: GET, Path: /whatever, Key: 475676e11...")
	fmt.Println()
	fmt.Println("example of offline log:")
	fmt.Println("Status: offline, Cache: HIT, Method: GET, Path: /whatever, Key: 475676e11...")
	fmt.Println()
	fmt.Println()

	t, err := url.Parse(target)
	if err != nil {
		log.Fatal(err)
	}

	http.HandleFunc("/offline", func(w http.ResponseWriter, r *http.Request) {
		online = false
		fmt.Fprintf(w, "offline\n")
	})

	http.HandleFunc("/online", func(w http.ResponseWriter, r *http.Request) {
		online = true
		fmt.Fprintf(w, "online\n")
	})

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		var err error
		if online {
			err = onlineHandler(w, r, t)
		} else {
			err = offlineHandler(w, r)
		}
		if err != nil {
			fmt.Println(err)
			w.WriteHeader(500)
			w.Write([]byte(err.Error()))
		}
	})

	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", port), nil))
}

func onlineHandler(w http.ResponseWriter, r *http.Request, t *url.URL) error {
	e := CacheEntry{
		Request: CacheRequest{
			Method: r.Method,
			Path:   r.URL.String(),
			Body:   reuse(r),
		},
		Response: CacheResponse{},
	}
	k := key(e.Request)
	fmt.Printf("Status: online, Method: %s, Path: %s, Key: %s\n", r.Method, r.URL, k)

	r.Host = t.Host
	r.URL.Scheme = t.Scheme
	r.URL.Host = t.Host
	r.RequestURI = ""

	res, err := http.DefaultClient.Do(r)
	if err != nil {
		return err
	}

	resb, err := io.ReadAll(res.Body)
	if err != nil {
		return err
	}
	res.Body = io.NopCloser(bytes.NewReader(resb))

	e.Response.Status = res.StatusCode
	e.Response.Header = res.Header
	e.Response.Body = resb

	eb, err := json.Marshal(e)
	if err != nil {
		return err
	}

	if res.StatusCode < 400 {
		err = os.WriteFile(path.Join(cache, k), eb, 0644)
		if err != nil {
			return err
		}
	}

	transmit(w, e.Response)
	return nil
}

func offlineHandler(w http.ResponseWriter, r *http.Request) error {
	k := key(CacheRequest{
		Method: r.Method,
		Path:   r.URL.String(),
		Body:   reuse(r),
	})

	b, err := os.ReadFile(path.Join(cache, k))
	if err != nil {
		if errors.Is(err, os.ErrNotExist) {
			code := 404
			fmt.Printf("Status: offline, Cache: MISS, Method: %s, Path: %s, Key: %s, Status: %d\n", r.Method, r.URL, k, code)
			w.WriteHeader(code)
			return nil
		}
		return err
	}

	var e CacheEntry
	err = json.Unmarshal(b, &e)
	if err != nil {
		return err
	}

	fmt.Printf("Status: offline, Cache: HIT, Method: %s, Path: %s, Key: %s, Status: %d\n", r.Method, r.URL, k, e.Response.Status)
	transmit(w, e.Response)
	return nil
}

func reuse(r *http.Request) []byte {
	b, _ := io.ReadAll(r.Body)
	r.Body = io.NopCloser(bytes.NewBuffer(b))
	return b
}

func transmit(w http.ResponseWriter, r CacheResponse) {
	for k, vv := range r.Header {
		for _, v := range vv {
			r.Header.Add(k, v)
		}
	}
	w.WriteHeader(r.Status)
	w.Write(r.Body)
}

func key(r CacheRequest) string {
	h := sha512.New()
	h.Write([]byte(r.Method))
	h.Write([]byte(r.Path))
	h.Write(r.Body)
	bs := h.Sum(nil)
	return fmt.Sprintf("%x", bs)
}

The cool thing here is that it can be used virtualy for anything and can be build for any platform and is just small single binary that can be used localy

Idea was that when you online - everything works as is, proxy just proxying requests to nuget, transmits response to cliend and saves it to cache

Whenever you become offline - proxy serves cache items only

Cache itself has no time to live and lives forewer

Whenever invalidation is needed and you online - just send request and new response will be written

If cache item must be removed - just delete its file

All cache files are plain json so can be easily traversed

Theoreticaly such proxy may check if we online or offline in background and switch mode automatically

Also it may perform some trick with hosts file and add itself instead of target but then there are chances of port binding issues so that needs some more investigation

Also interesting if there is a way to add some kind of system wide proxy for concrete hostname which will allow to use such proxy for things like npm, pip, whatever

NuCache

And the final solution vas to create dedicated script for offline

#!/usr/bin/env pwsh

function GetJson($url) {
  $urlPath = New-Object uri $url | Select-Object -ExpandProperty AbsolutePath -ErrorAction SilentlyContinue
  if (-not $urlPath) {
      return Invoke-RestMethod $url
  }

  $httpCachePath = "~/.nuget/nucache/http$urlPath"
  if (Test-Path $httpCachePath) {
      $lastWriteTime = Get-ChildItem $httpCachePath | Select-Object -ExpandProperty LastWriteTime
      $now = Get-Date
      $minutesSinceLastUpdate = ($now - $lastWriteTime).TotalMinutes
      if ($minutesSinceLastUpdate -lt 30) {
          return Get-Content $httpCachePath | ConvertFrom-Json
      }
  }

  $response = Invoke-RestMethod $url
  if ($response) {
      New-Item ([System.IO.Path]::GetDirectoryName($httpCachePath)) -ItemType Directory -ErrorAction SilentlyContinue | Out-Null
      $response | ConvertTo-Json -Depth 100 | Out-File -Encoding utf8 $httpCachePath
  }
  return $response
}

function NuCache($package, $version) {
  $registrationBaseUrls = GetJson 'https://api.nuget.org/v3/index.json' | Select-Object -ExpandProperty resources | Where-Object { $_.'@type'.StartsWith('RegistrationsBaseUrl') } | Select-Object -ExpandProperty '@id' -Unique
  $packageBaseAddress = GetJson 'https://api.nuget.org/v3/index.json' | Select-Object -ExpandProperty resources | Where-Object '@type' -EQ 'PackageBaseAddress/3.0.0' | Select-Object -ExpandProperty '@id'

  $package = $package.ToLower()

  if ($version -and $version -match '^\[\d+\.\d+\.\d+,\d+\.\d+\.\d+\]$') {
      $version = $version.Split(',')[0].Trim('[')
  }

  if (-not $version -or $version.Contains('*')) {
      $versions = GetJson "$($packageBaseAddress)$($package)/index.json" | Select-Object -ExpandProperty versions
      $last = $versions | Where-Object { -not $_.Contains('-') } | Select-Object -Last 1
      if (-not $last) {
          $last = $versions | Select-Object -Last 1
      }
      if ($version -and $version.Contains('*')) {
          $wildcard = $versions | Where-Object { $_ -match $version } | Select-Object -Last 1
          if ($wildcard) {
              $version = $wildcard
          } else {
              $version = $last
          }
      } else {
          $version = $last
      }
  }

  $file = "$($package).$($version).nupkg"
  $target = Join-Path -Path "~/.nuget/nucache/packages" -ChildPath $file

  $registration = $null
  foreach($registrationsBaseUrl in $registrationBaseUrls) {
      try {
          $registration = GetJson "$($registrationsBaseUrl)$($package)/$($version).json"
          break
      } catch {}
  }
  if (-not $registration) {
      $v = $versions | Where-Object { $_ -match [regex]::Replace($version, "\.\d+$", "") } | Select-Object -Last 1
      if (-not $v) {
          $v = $versions | Where-Object { $_ -match [regex]::Replace($version, "\.\d+\.\d+$", "") } | Select-Object -Last 1
      }
      if (-not $v) {
          $v = $versions | Where-Object { $_ -match [regex]::Replace($version, "\d+$", "") } | Select-Object -Last 1
      }
      if ($v) {
          NuCache $package $v
      } else {
          Write-Host "$package $version - not found"
      }
      return
  }

  $catalogEntries = GetJson $registration.catalogEntry

  $exists = Test-Path $target
  if ($exists) {
      $alg = $catalogEntries.packageHashAlgorithm
      $wantedHash = $catalogEntries.packageHash
      if ($alg -eq 'SHA512') {
          $hasher = [System.Security.Cryptography.SHA512]::Create()
      } elseif ($alg -eq 'SHA256') {
          $hasher = [System.Security.Cryptography.SHA256]::Create()
      } elseif ($alg -eq 'SHA1') {
          $hasher = [System.Security.Cryptography.SHA1]::Create()
      } elseif ($alg -eq 'MD5') {
          $hasher = [System.Security.Cryptography.MD5]::Create()
      } else {
          Write-Host "Unknown $alg for $package $version"
      }
      if ($hasher) {
          $stream = [System.IO.File]::OpenRead((Resolve-Path $target).Path)
          $bytes = $hasher.ComputeHash($stream)
          $actualHash = [System.Convert]::ToBase64String($bytes)
          $stream.Dispose()
          $hasher.Dispose()
          if ($actualHash -ne $wantedHash) {
              $exists = $false
          }
      }
  }

  if (-not $exists) {
      New-Item ([System.IO.Path]::GetDirectoryName($target)) -ItemType Directory -ErrorAction SilentlyContinue | Out-Null
      $ProgressPreference = 'SilentlyContinue'    # Subsequent calls do not display UI.
      Invoke-WebRequest -Uri ($registration.packageContent) -OutFile $target
      $ProgressPreference = 'Continue'            # Subsequent calls do display UI.
      Write-Host "$package $version - downloaded"
  } else {
      # Write-Host "$package $version - exists"
  }

  $deps = $catalogEntries | Select-Object -ExpandProperty dependencyGroups -ErrorAction SilentlyContinue | Where-Object dependencies -NE $null | Select-Object -ExpandProperty dependencies | Select-Object id, range -Unique | Select-Object @{n='Package';e={ $_.id.ToLower() }}, @{n='Version';e={ $_.range.Split(',')[0].TrimStart('[').TrimStart('(') }}
  foreach($d in $deps) {
      $df = "$($d.Package).$($d.Version).nupkg"
      $dt = Join-Path -Path "~/.nuget/nucache/packages" -ChildPath $df
      if (Test-Path $dt) {
          # Write-Host "$($d.Package) $($d.Version) - exists"
          continue
      }
      NuCache $d.Package $d.Version
  }
}




if ($args.Count -eq 0) {
  Write-Host "NUCACHE"
  Write-Host ""
  Write-Host "Commands and usage examples"
  Write-Host ""
  Write-Host "nucache status`n`tdisplay current online/offline status"
  Write-Host "nucache offline`n`tswitch to offline mode"
  Write-Host "nucache online`n`tswitch to online mode"
  Write-Host "nucache sync`n`titerates over ~/.nuget/packages/**/*.nupkg files and downloads missing ones and their dependencies"
  Write-Host "nucache /path/to/project`n`titerates over /path/to/project/**/*.csproj files and downloads packages with dependencies"
  Write-Host "nucache Newstonsoft.Json`n`tdownloads latest version of concrete package and its dependencies"
  Write-Host "nucache Newstonsoft.Json 12.0.3`n`tdownloads concrete version of concrete package and its dependencies"
  Write-Host "nucache add package Newstonsoft.Json`n`tdownloads latest version of concrete package and its dependencies"
  Write-Host "nucache add package Newstonsoft.Json --version 12.0.3`n`tdownloads concrete version of concrete package and its dependencies"
  Write-Host ""
  Write-Host "Workflows"
  Write-Host ""
  Write-Host "Always offline - switch mode to offline it will force you to use nucache always, so before doing dotnet add package you will need to call nucache first"
  Write-Host ""
  Write-Host "Online/offline - switch between modes, whenever you will be online call nucache sync and nucache /path/to/project to sync packages, when become offline just switch mode"

  Write-Host ""

  $count = (Get-ChildItem "$($env:HOME)/.nuget/nucache/packages/*.nupkg").Count
  Write-Host "Cached packages: $count"

  if (dotnet nuget list source | Where-Object { $_.Contains("nucache") }) {
      Write-Host "Current mode: online"
  } else {
      Write-Host "Current mode: offline"
  }

  Write-Host ""

  exit 0
}

if ($args.Count -eq 1 -and $args[0] -eq "status") {
  if (dotnet nuget list source | Where-Object { $_.Contains("nucache") }) {
      Write-Host "online"
  } else {
      Write-Host "offline"
  }
  exit 0
}

if ($args.Count -eq 1 -and $args[0] -eq "offline") {
  dotnet nuget remove source nuget.org
  dotnet nuget add source "$($env:HOME)/.nuget/nucache/packages" --name=nucache
  exit 0
}

if ($args.Count -eq 1 -and $args[0] -eq "online") {
  dotnet nuget remove source nucache
  dotnet nuget add source https://api.nuget.org/v3/index.json --name=nuget.org
  exit 0
}

if ($args.Count -eq 1 -and $args[0] -eq "sync") {
  foreach ($item in Get-ChildItem -Path "$($env:HOME)/.nuget/packages" -Recurse -Include *.nupkg -ErrorAction SilentlyContinue | Select-Object -ExpandProperty Name) {
      if (Test-Path "$($env:HOME)/.nuget/nucache/packages/$item") { continue }
      $item = $item.Replace('.nupkg', '')
      $idx = [regex]::Matches($item, "\.\d+\.\d+\.\d+") | Select-Object -ExpandProperty Index -First 1
      $package = $item.Substring(0, $idx)
      $version = $item.Substring($idx + 1)
      NuCache $package $version
  }
  exit 0
}

if ($args.Count -eq 1 -and (Test-Path $args[0])) {
  $projects = Get-ChildItem -Path $args[0] -Recurse -Depth 4 -Include '*.csproj'
  foreach($project in $projects) {
      $xml = [xml](Get-Content $project.FullName)
      foreach($reference in $xml.Project.ItemGroup.PackageReference) {
          if (-not $reference) { continue }
          $package = $reference.Include
          $version = $reference.Version
          NuCache $package $version
      }
  }
  exit 0
}

if ($args.Count -eq 1) {
  $package = $args[0]
  $version = $null
  NuCache $package $version
  exit 0
}

if ($args.Count -eq 2) {
  $package = $args[0]
  $version = $args[1]
  NuCache $package $version
  exit 0
}

if ($args.Count -eq 3 -and $args[0] -eq 'add' -and $args[1] -eq 'package') {
  $package = $args[2]
  $version = $null
  NuCache $package $version
  exit 0
}

if ($args.Count -eq 5 -and $args[0] -eq 'add' -and $args[1] -eq 'package' -and $args[3] -eq '--version') {
  $package = $args[2]
  $version = $args[4]
  NuCache $package $version
  exit 0
}

Workflows

Always offline - switch mode to offline it will force you to use nucache always, so before doing dotnet add package you will need to call nucache first

Online/offline - switch between modes, whenever you will be online call nucache sync and nucache /path/to/project to sync packages, when become offline just switch mode

Paths:

~/.nuget/nucache/bin/nucache - the script itself
~/.nuget/nucache/http - cache for http responses
~/.nuget/nucache/packages - saved packages